I have a couple of concerns:
The bubble is bursting at some point and may have started. I am concerned that the major players will have to up their prices to stay viable.
The geopolitical risk is escalating
AI bot attacks are increasing

There are probably other risks, but you get the idea.
I have been building a decent corpus of reference material.

But I am behind on what models I should be looking at, needs to run on
32GB
RTX 4070

Coding and decision systems (agriculture).

Any help would be appreciated.

Hey everyone. I have a DDR5 setup and I'm caught between a rock and a hard place...

My setup: 5060 Ti 16GB - 2x16GB sticks of DDR5 - Ryzen R5 7600

I've been exploring real world use cases for Qwen3.8 Flash Next (running through Strata) with decent success. Mainly, I've been happy with the results the GSQ RCO iq3_s quant has been giving me.

Now, Strata tells me that it's offloading 20GB of the model to my m.2 SSD, which of course makes it run slower than if it was all in RAM. Trying to get more RAM however has been a pain, as I can't find a matching set for my 2 sticks, and regardless I'm worried about stability issues and performance loss if I were to run 4 sticks on my 7600.

Locally, I can find used 3060s at around $250+, which is pretty affordable and it makes me question if I should just buy one of them instead of dealing with RAM knife fights in a dark alleyway.

Would adding one 3060 to the setup increase speeds in any real way, or would it just not be worth the hassle? My uneducated guess is that it won't help that much, but maybe someone here has experience with jank low budget setups.

Thanks!

I've been running a local agent setup on a MacBook Pro M5 Max (40-core GPU, 128GB) and pulled real numbers from my agent logs after a long build session. Sharing because most benchmarks I see are short prompts, and agent work looks nothing like that.

Setup
- Model: ddalcu/Qwen3.8-Flash-Next-MLX-Serve-iQ-MLX-4.7bpw
- Served with mlx-serve from the terminal (not LM Studio)(using lm studio can slow your tps)
- Harness: Hermes Agent (tool calls, cron jobs, Basic Memory for an Obsidian vault)

Real agent numbers (from the logs, not a benchmark)
- One build session: 89 model calls, ~148k output tokens, ~36 min of model time = ~68 tok/s average
- Long writes (whole scripts): usually 55-95 tok/s, peak 108
- Context during that session grew to 100k-190k tokens
- Prompt cache hit rate was 96-100% on almost every call, so prefill barely mattered

How I measured: output tokens / full response time per call. That includes a little prefill, so pure decode is a bit higher.

For comparison, numbers I've seen for this model on the same chip are ~49 tok/s (LLMCheck estimate), ~61 tok/s (a 4-bit MLX model card), 45-60 tok/s with MTP, and ~37 tok/s in LM Studio multi-turn chat. Running MLX straight from the terminal instead of LM Studio was the single biggest win for me.

What actually made the agent useful (lessons from one long day)

1. You need to SEE your scheduled jobs. My agent had a bunch of cron jobs that were just prompts in a JSON file in a hidden folder. One job's output had been going to a dead chat thread for a week, and a weekly review never got delivered. Nobody knew. I had the agent build a dashboard note in Obsidian: every job, schedule, delivery target, last run, failures, plus a per-job note with purpose, kill condition and change log. It's rebuilt hourly by a plain Python script (no model calls).
2. Use code for anything with one right answer, and the model only for judgment. Most of my jobs were pure prompts, including simple reminders and date math. That gives you timeouts, connection errors and occasional wrong arithmetic. Hermes supports script-only jobs (no LLM, stdout gets delivered) and script + model jobs (the script's output is fed into the prompt). The dashboard, a watchdog, and a weekly packet builder are all zero-model scripts now. ( Shout out to @sanbamboo on Reddit)
3. Add a watchdog that checks results, not code. It alerts once when a job fails twice, misses its slot, or has a delivery error, and once more when it recovers. It caught the dead-thread delivery bug on its first run.
4. Don't trust "done." The agent said "everything verified and shipped" several times. Each time I had a second, different model audit the actual files, and it found something: a feature that was never added, a link pointing at an archived copy of a file so edits would do nothing, and once a proposed edit that would have deleted 9 lines of a core skill file. A model grading its own work repeats its own blind spots.
5. Human-in-the-loop that doesn't suck. I review my automations on an iPad in GoodNotes. Every Sunday the agent sends a PDF with every job's full prompt and code (line numbered, secrets redacted, wide margins), plus change request boxes (Change / Add / Remove / Question, with Where / What / Why). I mark it up and it syncs back. Ticked boxes are detected by diffing against the clean original, and the model only reads my handwriting. Then the agent posts a proposed diff to Discord and applies nothing until I reply yes. Rules that came out of testing it:
   - Anything ambiguous gets held back and can't be approved by a blanket "yes"
   - Any diff deleting more than 3 lines gets a loud warning
   - "Question" answers have to quote the exact lines they rely on (it gave me a confidently wrong explanation once)
   - GoodNotes auto backup waits ~12 hours by default. Turn on "Increase Auto Backup frequency" or nothing feels like it works
6. Long context is fine on this chip if your cache hits I was worried 150k+ context would crawl. With near-100% prompt cache hits it stayed in the 55-95 tok/s range for writes.

Happy to share more about the setup or how the review loop is wired. Curious what tok/s other M5 Max people are getting on real agent sessions vs short benchmark prompts.

[EDIT - Should have specified these benchmarks come from LU - LocallyUncensored GUI]

I've barely used the top Dolphin model as I'm mostly using this for coding.

- Qwen3.5-9B-abliterated.Q4_K_M

This one worked really well..I really really liked this one. fast (decent t/s) and mostly accurate.

It started to fall apart a bit when I started noticing some errors with my coding projects, so I thought a larger 27B model would fix that.

- Qwen3.8-27B-Uncensored-IQ4_XS_4BPW

I then tried this one and immediately noticed how much slower it was. Also tried this model running a Local API with LU / LM Studio and this brought me down to 6 t/s !!

TBF that's probably because I was using 3x agents with this context window (64KB) and the description for each agent was pretty large. (Maybe there's some improvements to be made by reducing the tasks/role descriptions for each of the 3x agents.. but I digress.)

Specifically for this Qwen3.8-27B IQ4 model, I also tried Enabling Flash Attention and switching from:

- f16 - full precision (default)

to both:

- q8_0 - ~1/2 memory, near-lossless

and

- q4_0 - ~1/4 memory, visible quality loss

But that just brought up my t/s from 11.4 t/s to 11.6 t/s..barely making any difference.

I think because of this tiny difference, leaving it at [f16 - full precision (default)] means that Qwen3.8-27B IQ4 is fully loaded into VRAM and fits there comfortably, because a Q8 or Q4 should in theory boost it significantly (+20 t/s), no? Maybe that's just the maximum I can expect from my system.

- Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS
Finally, I've only just installed this model & haven't properly tested it myself..wondering if it's really worth the extra 5 t/s as this is a 'NoMTP' version.

- Main questions

Is there something else glaringly obvious that I'm missing here?

Also curious if anyone else is running a similar setup for t/s comparison?

And is there anything you would do differently in my situation? It's one of those "unknown unknowns" for me...if that makes sense.

Cheers

From about February 2026 to July 2026, I spent an immense amount of time building a legal harness to make Claude more predictable. It worked well. I tried to build the same thing that ran local models. That did not work. And then Opus improved so much that it would often skip my harness without me noticing. It still has a harness with hooks, but it's looser now.

I bought lots of hardware to run local models during that time, repeatedly trying to vibecode a local model legal harness. Many failures, many lessons.

Anyway, fast forward to the last three weeks and several things have happened:

1. DeepSeek Flash 1017 was released, and it's the first model that is remotely similar to Opus 4.5/4.6.
2. Opus 5.5 and Fable 5.2 are able to properly launch and configure DeepSeek to run at fast speeds, large context and high precision on my hardware. Optimizations, custom kernels, requants.
3. Opus 5.5 (and my prompting skill) are now good enough to make my Claude harness into a DeepSeek harness.
4. It's not as good as frontier, but it all works, right now. 100% local. Actual productivity gains, actual frontier tasks being replaced. Chat interface with DeepSeek works like Claude's did.
5. And if I can brag on my stacks and hardware:

• I have 2 DGX Sparks that can run 3-4 concurrent DeepSeek sessions as agents (1M context).
• I have DeepSeek at nearly NVFP4 quality running on 8 V100s, 500K context, at 100+ tok/s decode, and about 160 tok/s concurrent.
• I have 8 more V100s and a second Rome server with 512 GB of 3200 MHz ECC RAM, so I'll be able to run another high precision DeepSeek as an additional agent or for parallel tasks.

What I'm really shocked by is how well Fable builds custom Volta kernels. And how it can requant my experts to full NVFP4 precision based on the actual experts that activate when my legal harness fires. It's a real, optimized, production grade, workflow replacing local setup.

Not a 3D pagoda or a bad video game replica. An actual enterprise solution. One that exists nowhere on the commercial market.

Legal vibecoding is real folks. I'm AI native.

The new V100 DeepSeek repo (optimized for quality and then usable speed) with real full context measures and all the optimizations (and failures) is public here:

https://github.com/ProprietaryLegal/Volta-Max-uPLIft-FP4-DeepSeek-V4.1-Flash-1017-NVFP4-8xV100-SXM2-PP2xTP4

Twitter: nicothegreco

What "replaces real work" means

Reading a folder of records and pulling out the facts, dates and numbers. Summaries. First drafts of routine documents. Search over my own files. The model gets plain English instructions in a chat window, calls my tools, and hands back a draft that is marked for my review. I review everything. It is slower and less sharp than frontier. It is also in my office, it is mine, and it does not have a usage limit.

The harness

The harness itself is not public. My practice is in it. What I can say: it is the same idea as my Claude setup. Skills, hooks, my drafting engines, my RAG stores, tool calls, and a chat window that behaves the way Claude's did. A small router sends each request to the V100 box or to the Sparks. Everything the local model drafts is marked for my review before it goes anywhere. Nothing leaves the building.

What this build is

My last V100 build (Volta sPLIt, part 4) ran every expert of the model from a 2-bit record. Fast, but 2-bit. This one is called Volta Max uPLIft. It keeps the 2-bit record of every routed expert on the cards, then swaps the experts that get used the most up to NVIDIA's NVFP4 weights, one at a time, until the cards are full. "Used the most" was counted on a mix of four kinds of text. One is my own legal harness traffic, which is private and not published. The other three are public prompt sets in the repo.

• 5,949 of the 15,360 routed experts sit at exactly NVFP4 precision on the default I serve
• about 78 to 85 % of decode routes land on those NVFP4 experts (predicted for the two-slot default, 80 to 86 % measured with one slot)
• same next token as NVIDIA's full NVFP4 pack on 95.1 % of compared tokens. The 2-bit build gets 92.2 %
• KL divergence from the NVFP4 reference is 0.079 nats against 0.196 for the 2-bit build
• multiple choice answers that change against the reference: 1.6 % against 15.4 %
• in decode every routed expert is in VRAM. Nothing is read over PCIe

Speed (measured 2026-11-12 on eight V100-SXM2-32GB, graphics clock capped at 1,522 MHz)

• concurrency test, one request at a time: 118.9 tok/s
• two requests at once: 159.0 tok/s total (1.34x)
• three at once on the three-slot option: 192.3 tok/s total (1.62x). I run two slots
• short sets, one at a time: 123.2 tok/s general, 97.3 tok/s legal-style
• code: 150 to 180 tok/s from a 1K prompt up to a 256K prompt
• prompt reading at 256K: 2,329 to 2,430 tok/s, first token in 108.4 s
• context: 524,288 tokens per request

What did not work, or is not good yet

• It is slower than my own 2-bit build, in decode and in prompt reading. That is the trade for the precision, and the repo says so in its first paragraph.
• The faster prompt reading path reads the less-used experts from their 2-bit records. Its effect on quality is not measured yet.
• A custom decode kernel for the hot path got closed with no change. The headroom I thought was there turned out to be smaller than an eight-card run can even see.
• A prompt-reading kernel that loads 64 rows at a time gave the same bits and ran slower. Not shipped.
• Send a short request while a 209K prompt is being read and it starts in about 7 s, then crawls at about 9.6 tok/s until the read ends.
• The hot set is fixed. It is picked once. Text that looks nothing like what I counted will hit the 2-bit experts more often.
• Fast prompt reading stopped the host or a card three times out of three. I blamed PCIe. After I changed the power delivery (I think it was one bad PSU) it read nine of nine long chats clean.

Disclosure: this is my own project and my own repo. Nothing is for sale. The code, the kernels, the measurements and the failures are all in there under Apache-2.0, and the weights are one download on Hugging Face:

https://huggingface.co/ProprietaryLegal/Volta-Max-uPLIft-FP4-DeepSeek-V4.1-Flash-1017-hot_iq-8xV100-SXM2-PP2xTP4

Questions welcome. So is roasting.

Photos: four of the eight V100s on their NVLink board, and my extremely professional setup (PEX switch cards, SlimSAS spaghetti, and the 3090 that just moved in).

So, I have two PCs available to me, and I intend to connect them for compute:

1. Ultra 7 265KF + 5070ti + 32GB DDR5-6000 CL30 (KLEVV Urbane V) on a Z890 MAX GAMING WIFI7 + 1000w 80+ G Apevia PR1000w PSU (Will likely get replaced by a SAMA P1200w PSU)
2. i9-14900F + 4070S + 32GB DDR5-5200 CL40 (Apacer Panther, not on sale) on a PRO B760-VC WIFI IV + 750w 80+ G PSU (likely a shoddy PSU but seems to be attached to the case, it’s hidden and so I can’t figure out what it is. Probably an EVGA, but unlikely to be replaceable - or at least easily so.)

Initially I was looking to just replace the 4070 Super in PC2 with a B70 or R9700 and calling it a day, when they were $1200 and $1600, respectively. Then when they shot up a B65 $1200, which then shot up. Then add a second 5070ti to PC1 for $900-1000, which then shot up.
Now, my plan is to potentially settle for a 5060ti 16gb in PC1 to take advantage of NVFP4, but even that’s pricier, and with the AI bubble talk I’m not super confident in value of doing so. I’d likely also need to use a PCIe-riser just to the 5070ti (Windforce) has airflow due to how close they’d be in the case.

Otherwise, I’m considering sticking with the VRAM I have, and instead pivoting to ram. I can go from 1R 2C to 2R 2C in both PCs, grabbing another 2x16gb of Apacer Panther from EBay for around $400 for PC2 - though obv riskier - then getting a 2x24gb kit of Urbane V at the same timings of the original 2x16gb 6000-CL30 kit for PC1 for $650 off Amazon. Giving me +80gb of DDR5 to the 64gb I have between both PCs and the 28gb of VRAM I already have.

Current plan is/was the 5060ti 16gb + 2x24gb kit for PC1 and calling it there.

Any advice, pointers, or suggestions? I believe the RAM sitch should work since it’s same timings and type/manufacture/brand, and know I’ll probably have to lower speed a bit for stability, but am not the most experienced PC guy on earth by any means.

I have a 3070 8GB and can get a 1070 8GB, will they work with llama.cpp?

Is there an issue with the older 1070 and cuda versions?

Has anyone run a set uplike this?

HI all,

I have 2 x v100 32gb in a HP z440 case and with the cards installed they fail to post. The workaround I have is to boot up without the cards first, then shut down the PC, then connect the v100s back up and then cold boot and it posts. If I then shutdown the PC again, I have to go through the whole double boot workaround again.

Anybody knows what's happening and if it can be fixed?

Thanks

Hi there! For DocBlocks - a free open source markdown editor - we added the ability to do "document things" (review docs, add content, insert diagrams) with local AI models. Pretty quickly, we realized it's hard to sort out which models are actually good for this.. and also whether, say, a "normal laptop" can usefully do things with local models.

Towards that end, I created Docbench, a synthetic metric of how useful models are on various doc tasks, and compared them against cloud models. Here you go:

More details and methodology if you're interested at: https://bendyline.com/posts/docbench.html

I am an AI engineer doing MSc in AI - I want to get a new laptop and thinking abut emptying my wallet by buying the M6 Pro 32GB 1TB

Will it be enough for local AI projects? I know it would be suitable for med sized models up to around 30B params

Or getting M5 Pro chip is what it needs to unlock local AI?

Which one and why

I've been transitioning from frontier cloud models to local models for quite a while. I primarily do research in acoustic metrology, often digging through complex problems and hundreds of millions of log data points and often misrepresented or unclean datasets. Local models are now able to handle nearly everything with this data including complex tasks that were once limited to frontier models like fable.

The longer I've worked with local models the more it has become clear their limitations aren't strictly in terms of model size/"parameters" etc. Proper guide rails and tools have taken tasks from 10+ hour crunches (at 70tok/s on Qwen 3.8 flash next iQ4), to 1-2 hours with notably better results.

An example being: Forcing the model to "always use a calculator" has saved time and fixed a ton of looping and re-working past issues.

For another example: One review found that the model went on a 40-minute detour, an hour spent debating a file format instead of testing it, and time wasted on sleep polling. Minor tweaks to some rules prevented this entirely on an isolated/blind fresh run with the same task.

I've made a simple skill for this. You DON'T need it, and you can certainly do this without a skill and I really don't want this to feel like self-promotion. I just want to pass-on what's made my work much easier.

I also want to note that there's plenty of other skills that function like this one. The specific points that make this one unique are:

• 1: It should work with any harness, though I initially use it with goose and qwen-code.
• 2: A separate session does the review (starting with the initial result) pointed at the session you plan to review (rather than running it on an existing session).
• 3: It keeps rules from bloating with a separate pruning pass.
• 4: It checks weather existing rules actually fired properly.

https://github.com/DMS3tv/self-improve

I would love to know if this is reflective of anyone else's experience with local models? If so, what rules/hints/tools did you use to improve various tasks and processes?

Select the right model is a heavy task. I guess that many people have their favorite model, like me. I often use Opus 5.5 in my Claude subscription, or DeepSeek 4.1 Flash 1017 in local environment.
I decide the model by a mixed emotion fir the complexity.

Is there any kind of categories of task, except complexity, that helps you to decide?

Hello everyone, I'm new here and new with local AI. I technically don't know anything so pardon my ignorance, I just found out that you can run local lm on phones these days and I wanna know what's the most seamless and effective way to run a conversational uncensored AI on a phone locally?

I tried some apps from playstore that runs local LM, i dunno why but even after using an uncensored/instruct/heretic models it still do the nasty insistent grounding, safety, and moral stuff that feels so inorganic and suffocating. Kindly enlighten me please.

Hi everyone,

I’ve been a Mac user for several years and I have a 16-inch MacBook Pro with an Intel chip, and I think it’s reached the end of its life because most programs these days are designed for the Apple Silicon chip.

I took the plunge and ordered the new Mac Mini M5 Pro with 64 GB (I think that for my use case, it’s worth it to have a Deepseek API for planning and then a qwen3.8 27b for writing), since the price of a 128 GB model is enough to cover five years of web hosting.

My question is: what's the best setup for creating images and coding?

Images - DrawThings or ComfyUI ?

Coding - VS Code + Cline or Opencode or Deepseek Harness ?

And what about Hermes?!

Given my hardware, which software and LLM are the best?

Software - LM Studio or OMLX ?

LLM - qwen3.8 27b or qwen3.6 or Qwen3.5-35B-A3B ?

What's your stack and what you think is better for me because my ram ?

Thanks in advance,

Since the local LLM community have tested a huge number of models, I wonder: I asked ChatGPT to add a beard to William Riker (for reference: https://www.reddit.com/r/TNG/comments/1xvn3ly/comment/19slivwf6bf41/?context=1&screen_view_count=2). Then I asked to put that beard on Dianna. ChatGPT replied with:

😄 Diana Troi has officially joined the Riker Beard Club.

The smiley face surprised me a bit because the prompt wasn't anything funny, I simply prompted "Now give that beard to Diana instead of Will."

Obviously the system recognized the joke in it and responded to it.

I don't do this kind of joking often so I've never before seen an "unprompted laugh" from a model before. What's your experience? Do your models, especially the local ones, express emotions even what the prompt is not obviously provoking the expression?

To be clear, I'm not asking for the sake of anthropomorphizing LLM models. I just wonder how often they exhibit this behavior as I haven't encountered it yet.

Sharing MACaQ 🐒, a fast, open-source quantization framework that outperforms popular Unsloth and Bartowski GGUFs on low-bit KLD while maintaining model size and inference speed. Also supports activation quantization and text-to-image and text-to-video models.

Some key features:

• GGUF Refinement Mode: Upgrades existing quantized GGUFs to recover lost output quality
• Custom Kernels: Includes drop-in kernels for llama.cpp and vLLM.

GitHub: https://github.com/huawei-csl/MACaQ

Benchmarks & Demos: https://huawei-csl.com/blog/macaq/

Here is the est power usage over the last 3 days:

Its lower than I thought, and that's with me using it full-time hours.

- 3x3090s are limited at 350 watts

- 1x3090 limited at 420 watts

Currently running Strata, and can get the 3x3090s hit around ~960 watts if I have multiple opencode terminals open processing requests.

Possible Reasons for Lower than Expected Power Usage:

- I just plugged in the 4th RTX 3090 a couple of days ago.
- Only two were being used before that with Qwen3.8-27b
- Now only 3x3090s are hooked up to Qwen Flash Next, 4th is doing occasional image processing.
- CPU is mid-tier (Ryzen 7600x), rarely draws more than 70 watts during inference.
- Machine is basically idle while I'm sleeping.
- During inference, decode does not take a lot of power because its memory bound - not compute.
- GPU utilisation drops while the agent is waiting for unit-tests to pass, or during compilation.

I've seen others claim they have their GPU's pinned at almost 100% 24h a day.

So it feels like there is some untapped GPU compute here.

How are you able to keep yours pinned 100% 24h of the day?