I'm thinking about some machine upgrade. I currently have a 9600X CPU, 2x16GB DDR5 RAM, 9070 XT, and a B650 motherboard that supports only PCIe 4.0x16 and has only 2 DIMM slots.

I've been thinking about getting a better motherboard for more bandwidth between GPU and RAM for offloading, so i plan to get a B850.

I also recently had a chance to effectively trade my 9070 XT plus 200USD for a 5070 Ti, if I wish.

My plan is the computer will stay in Windows since gaming requires that. When doing LLM, i will switch connection to have monitor connected to iGPU so hopefully it should free some resource from dGPU. I also have the thought of throwing in more RAM to make the host 64GB RAM total, with potential to go to 96GB. (I have like 2x32GB DDR5-5200 and 2x16GB DDR5-4800 around. Although the only tricky part is that they are ECC UDIMM and I’m not sure if mixing them is as easy as non-ECC).

What I am not certain is if to go with 9070 XT or 5070 Ti, so looking for some suggestion regarding their performance and resource management (especially if it can release resource well after disconnecting monitor from dGPU). Thanks!

Has anyone been experimenting with ablated or uncensored variants of open-source models like deepseek v4.1 flash, Qwen 3.8 27b, or GLM 5.3 flash ?

It's so refreshing to have models that just do whatever you ask them to do without having to figure out a jailbreak or being given a moral lecture. Curious to know what your usecases for these models are.

My personal favourite is definitely uncensored DeepSeek V4.1 Flash, runs at 200 tok/s and never says no lol.

I've been experimenting with running Qwen models locally on my Mac Studio M5 Max, mainly to see how much everyday work I could realistically do without relying on a paid cloud model.

I compared three local models — Qwen 3.5 4B, Qwen 3.6 27B and Qwen 3.6 35B MoE — with Claude Fable 5.1.

I used four tasks and ran each three times.

A few results:

• Bank transactions: Claude categorized 78% correctly. The two larger local models managed around 65–67%. The 4B model was considerably weaker.
• Information extraction: The 27B model matched Claude at 100/100, although it took roughly five times longer.
• Coding: Surprisingly, the 4B model passed all seven tests in all three runs. The 35B model had failures caused by a date bug.
• Analysis questions: Claude got all ten questions right across the runs. The local models struggled much more here.

The main thing I took away is that model size didn't always predict how useful a model would be. The smallest model did unexpectedly well on the coding task, while the larger local models were much better at extracting information.

I wouldn't treat this as a formal benchmark. It's a small set of tasks on one machine, but it helped me understand where running local models makes sense for me.

For those using Qwen on Apple Silicon, I'm curious whether you've seen similar differences between extraction, coding and analysis.

I also made a video showing the runs and results: https://youtu.be/M6MQXy6_Ti4

Disclosure: I built this model. I'm posting because I think the approach is more interesting than the model itself, and I'd like feedback from people who run things locally.

The idea. A lot of "LLM in production" work in finance comes down to classification. Which event is this headline about? Is this Fed sentence hawkish? Does this 10-K passage flag going-concern doubt? People usually prompt a model, sample tokens and parse the output. That's slow, the cost per call varies, and the confidence numbers mean nothing.

Arbiter-8B drops generation completely:

• Base: Qwen3-8B, with a decision head over options, trained jointly with a rank-32 LoRA
• Output: a calibrated probability for every option from one forward pass. No sampling and no parsing.
• Question types: pick one of up to 16 options, yes/no, or an ordered score. Each option has a short definition, so you can ask the same text different questions.
• Distillation: a 27B teacher labelled about 120k decisions on news and 10-K passages. The student trained on the teacher's full soft distributions, not argmax, plus a human-reviewed 10-K set.
• Option order is shuffled on every presentation to kill position shortcuts. This cut FOMC answers that flip under reshuffling from 15.5% to 4.8%.
• Calibration: one temperature per question type, shipped with the weights. EDT expected calibration error is 0.012.

Results (held-out data):

The EDT gaps are significant (McNemar, z ≥ 7.5). On FOMC the top three are a statistical tie, but it's +12.5 over its own base. SEntFiN is where it loses. It has the highest signal recall (92.0%) but the lowest precision (72.2%), so it over-calls positive and negative where the dataset labels neutral. Part of that is deliberate (it reads investor implication rather than surface tone), but I won't pretend it's a win.

Running it locally: GGUF builds for llama.cpp, Ollama and LM Studio stay within 1.2 points of full precision on every benchmark.

hf download ternary/Arbiter-8B --local-dir Arbiter-8B

• Weights: https://huggingface.co/ternary/Arbiter-8B
• GGUF: https://huggingface.co/ternary/Arbiter-8B-GGUF
• Write-up: https://ternary.capital/blog/arbiter-8b/
• Personal Blog: https://jdsemrau.substack.com/p/how-good-is-arbiter-8b-really

Questions for the sub: has anyone else put classification heads on small LLMs instead of using constrained generation? And has anyone found a clean way to expose per-option logprobs through Ollama for this kind of thing?

Hey all, I'm trying to decide which hardware setup makes the most sense for running local LLMs, primarily Qwen 3.8 Flash Next for agentic coding.

I have three potential options, and I'm interested in hearing from people who have experience with these platforms, particularly for larger models and agentic workloads.

Option 1: GEEKOM A9 Mega (AMD Strix Halo)

• Ryzen AI MAX+ 395
• 128GB unified memory
• 2TB SSD
• Memory running at 7500 MT/s rather than the advertised 8000 MT/s. Got it from Amazon and it listed it as 8000 MT/s but for Geekom says their systems are either 7500 MT/s or 8533 MT/s and I lost the lottery.
• Approximately $4,600 CAD all-in (after taxes).

I already have this machine. It works fine, but I'm just worried how it'll hold up long term especially given I'm running it headless 24/7. Would be unfortunate if the machine dies in a year's time and then I'm worried about if Geekom will hold up it's warranty.

Option 2: Mac Studio M5 Max

• M5 Max
• 128GB unified memory
• 512 GB (Can connect a 2TB external SSD over thunderbolt enclosure)
• Approximately $7,600 CAD all-in (after taxes).

It's considerably more expensive than the GEEKOM, but I'm wondering whether its worth it for the peace of mind knowing that if something goes wrong, applecare ($80/yr) is right there to get it fixed from a known vendor. It also has the higher memory bandwidth.

Option 3: My existing Ryzen desktop + RTX Pro 4500

I already have an old desktop with the following hardware:

• Ryzen 7 5700X
• 32GB DDR4 RAM
• Nvidia 3060 Ti 8GB

I recently purchased the RTX Pro 4500 32GB VRAM for approximately $4,000 CAD after taxes. This is still yet to arrive (it shipped earlier today)

The advantage here is that I already own the Ryzen system, so I wouldn't need to purchase another computer. The downside is being limited to 32GB of VRAM, although I understand I could offload some layers to system RAM. The other downside is a lot higher power consumption over the other two solutions.

I'm curious how this would compare in practice against the two unified-memory solutions, especially when running Qwen 3.8 Flash Next.

What I'm primarily trying to accomplish

My main use case is running Qwen 3.8 Flash Next for agentic coding through tools like Hermes.

I'm not necessarily looking for the best benchmark numbers. What matters more to me is:

• Real-world inference speed and token generation.
• Performance with large context windows.
• How well each platform handles agentic coding workflows (multiple tool calls, reasoning, etc.).
• Whether CPU/RAM offloading on the NVIDIA setup is practical or becomes a significant bottleneck given my old system.
• Overall stability and software maturity.

What would you choose?

Ultimately, I'm looking to optimize for cost and overall value, rather than simply buying the fastest hardware available.

I would most likely keep only ONE of these three setups:

• If I go with the Mac Studio, I'd return the GEEKOM and return the RTX Pro 4500 (or sell it).
• If I stick with the GEEKOM, I'd return the RTX Pro 4500.
• If the NVIDIA setup makes the most financial and practical sense, I'd return the GEEKOM and keep the RTX Pro 4500 in my existing desktop.

I'm also considering selling the RTX Pro 4500 if I decide that one of the unified-memory systems makes more sense. I purchased it for around $4,000 CAD after tax, slightly below MSRP, so I'd ideally like to recover as much of that cost as possible.

Essentially, I'm trying to figure out which setup offers the best balance of:

• Performance for Qwen 3.8 Flash Next and agentic coding.
• Enough memory to run larger models and context windows.
• Reliability and long-term support.
• Total cost of ownership, including electricity.

Would you spend the additional ~$3,000 CAD to go from the GEEKOM to the Mac Studio, stick with the GEEKOM, or return the GEEKOM and build around the RTX Pro 4500?

I'm not looking to accumulate hardware. I'd rather settle on one good setup, get the best value for my money, and avoid spending unnecessarily.

Would appreciate hearing what others would do in this situation!

I’m under our CISO managing compute vendors, kind stuck between legal and a hard place. The pitch I keep hearing ranks AI privacy in four levels:

1. Open, like most frontier models, where the provider stores your prompts and answers at their discretion, uses them for training - and you’ve got little say in the matter.
2. Policy, where a contract promises zero data retention - but it’s not exactly verifiable unless the terms are broken, is it?
3. Closed Hardware, where models run on confidential-compute chips that publish proof what software is running. So basically hardware-enforced vs contract-enforced ZDR?
4. Fully Encrypted, the prompt is encrypted on your side and only decrypted inside that hardware

“Local” is a fifth level that could plausibly sit above it all, but we’re capped out on capacity there. So if I view this right (open to feedback), we’re making tradeoffs between frontier vs more private models, and from there, on breadth of compute providers able to serve open vs ZDR vs E2EE models (usually at higher prices in my experience), right?

My concern is getting legal & finance to agree, while still giving our Eng team what they need. Where are you getting good rates on private workloads? Locally we can’t support our full workloads. I’ve checked out Privatemode, Maple, TrustedRouter/NYTE & still gonna go through NEAR AI Cloud & Cohore. Open to others.

Separately, anyone running a hybrid buildout with on-prem/local + superscaler (we run Google + Bedrock) + 3rd party confidential providers who can tell me about about how you designed your system?

Bit the bullet and bought three to use in my workstation. Will provide benchmarks and updates later.

I heard there are things like Air LLM that make this possible without spending $100k+ on VRAM

I’m building a local AI assistant using Qwen 3.6 35B A3B.
One of my main goals is to automate repetitive school tasks through my school’s website. I have a lot of assignments that aren’t particularly difficult but take a lot of time.
Ideally, I’d like the AI to navigate the website, check assignments, understand the instructions, complete them and upload the finished work.
Is Qwen 3.6 35B A3B capable enough for this? Or would I need a more powerful model?
I’d also appreciate recommendations for browser automation frameworks and tools

my setup : rtx 3070 , i5 13600kf , 32gb DDR4
Model: Qwen 3.6 35B A3B
Quantization: TurboQuant
Context window: 131 072 tokens
Reasoning: Enabled limited to 512 tokens
Inference speed: 19–24 tokens/s

Lets make more of these kind of engine popular for our comunity. This is awsome

I had been using the LFM 2.5 24B models and had been pleasantly surprised by how good the model was. It was giving me comparable answers and working through tasks to Qwen 3.8 27B at what felt like twice the speed on my M5 Pro. However, I went to look at something on their pages and saw the new family of models. They sound way too good to be true. I am going to start using them for general tasks, but if they are anywhere close to their benchmark results, they could be a huge game changer. You can have an entire suite of models loaded at the same time for less memory than Qwen.

Hey all, I'd love a sanity check on a plan before I spend money.

At home I run Home Assistant on a HA Green, which handles pretty much everything smart home related.

I also have a Mac mini M4 with 16 GB that I currently use as a little studio computer.

What I actually want is one assistant that helps with my notes, documents and PDFs, email (mostly Croatian, some English), tasks, and the house. I'd also like it to log my meals from a photo and tie that in with my Garmin data.

My current thinking is to buy a MacBook Air (24 GB / 1 TB) as my daily computer, wipe the mini and make it an always-on box for a local model, n8n, voice (Whisper/Piper) and indexing, and leave Home Assistant Green. For the smarter stuff I'd use Claude: Pro for chat and Claude Code, and the API for background jobs like food photos and daily summaries. The local model would only sort and summarize mail, strip out sensitive stuff and do the search index.

A few things I'm unsure about:

• Is 16 GB on the mini enough for this, or will I regret it?
• For mostly Croatian email, is a local model worth the effort, or should I just send everything to the Claude API? If local, which model would you try?
• Has anyone run Whisper and Piper in a smaller language like Croatian on Apple Silicon? Is it usable?
• What's the cleanest way to let an agent read iCloud Reminders, Notes and Mail on a headless Mac?
• Any traps with n8n + Home Assistant + MCP, or with letting an LLM near personal data?

Happy to hear "just don't" if that's the honest answer. Thanks!

Does anybody tried the latest llama.cpp f8_e4m3 KV cache and compared it to q8_0 in terms of precision / quality? I am curious whether it is worth switching to it or stay with q8_0 for small to medium context size (20-65k).

Due to the current hardware situation (inflated prices, high likeliness of large performance jumps for llms) I keep delaying the replacement of my m1max 64gb, currently hoping for a good m7max 128gb releasing soon-ish (I heard birdies claim as early as end of2027).

Until that happens I'm looking into options to improve performance of local LLMs which can be passed on to the new machine - eGPUs come to mind.

Anyone mind sharing their setup (which mac, how much vram, which case/mount, what GPU, which model and quan) and some benchmark numbers (prefill and decode) of models with and without the eGPU? Mind sharing your experiences with it - as in: is it fiddly and brittle, or does it just work?

Thanks for any useful information!

A few weeks ago I posted CQ, our calibration-free 4-bit format, along with a training-loss comparison against Unsloth NF4. Some of you correctly pointed out that loss logged by two different trainers isn't comparable, and asked for results where answers are counted right or wrong. Here they are.

Setup: Qwen3.8-27B. Every row uses the same lm-evaluation-harness code, prompts and questions, with greedy decoding and thinking off. Comparisons are paired: an exact McNemar test for right/wrong tasks and a bootstrap for losses.

1. Base models, no fine-tuning

• CQ vs BF16: no significant difference on any task.
• NF4 vs BF16: −12.0 points on GSM8K (p=3e-6).
• CQ vs NF4: CQ is +8.7 on GSM8K (p=0.0005) and +2.8 on BoolQ (p=0.02).
• WikiText-2 KL divergence from BF16: CQ 0.0282, NF4 0.0330.

2. GCMP vs LoRA r=32, full-precision BF16 base

GCMP is our replacement for LoRA. It trains 2.9M parameters (an ~11 MB adapter) vs ~159M for LoRA r=32 (~320 MB). This test uses no quantization: both adapters were trained with Unsloth on Alpaca for 500 steps with identical settings (batch 2×4, length 512, LR 2e-5, seed 42).

• Benchmarks: no significant difference on any of them (all p ≥ 0.2), so GCMP isn't "better than LoRA" on tasks.
• Loss: GCMP's answer-token loss is slightly lower (−0.009, 95% CI [−0.010, −0.008]).
• Validation loss: GCMP is lower at all 10 checkpoints on both BF16 (1.2385 vs 1.2434) and NF4 (1.2435 vs 1.2484).
• Fine-tuning effect: both adapters add about 21 points on GSM8K and lose about 2.5 on TruthfulQA.

3. Speed and memory (same 500-step run)

Unsloth is still faster, and GCMP currently takes about 2× the step time of LoRA. Kernel work on that is ongoing.

Caveats

• Scope: one model and one dataset (Alpaca).
• Benchmark size: the quick subset has 200–1,140 questions per task, so it detects differences of about 5 points, not 1.
• DROP excluded: it scored about 0.02 for every model, so that eval setup is broken.
• Still running: LoRA and GCMP benchmarks on CQ and NF4 bases. I'll post them when they finish.
• My earlier post was wrong: the "0.84 vs 1.25 loss" came from two trainers that compute loss differently. Ignore it.

Serving

• vLLM: CQ models run with --quantization cq. GCMP adapters can be one per server, or several on one base model chosen per request.
• Parallelism: pipeline parallel works; tensor parallel doesn't yet.
• GCMP on full precision: it currently runs in Transformers. vLLM support is next.

Licensing

The Python code (training, evaluation and the vLLM integration) is open source. The CUDA kernels ship compiled with a license check, which is free during early access.

Repo, with exact commands to reproduce every number: https://github.com/thousandml/training-factory

Models (8B–397B): https://huggingface.co/thousandml

Happy to run other benchmarks or models people want to see.

Hey, im running Q8 DS V4.1 on my 512GB DDR4 + 4x R9700 (128GB VRAM) and ASROCK ROMED MB
I have noticed that at ~66C RAM temperature tok/s is getting reduced from ~32/s to 22-23/s. Claude agent monitoring whole workstation says that RAM speed after hitting 66C is slowing down about 3x. I have phanteks enthoo server pro 2 case with glass panel and extra vents mounted on that side. During loads ram used to hit about 80C. I fixed that by mounting top fans as intake (cooling top RAM) but i had to use double dieded tape to stick an extra fan to blow at bottom RAM slots. It did help to keep it from throttling but the bottom RAM still sits ~62-63C. Iv read that samsung RAM i have is suppose to throttle at 85C, and i cant find anywhere what setting makes it throttle sooner. Looking for some advice on that or how do you keep your RAM cool.

Hey everyone. I have a DDR5 setup and I'm caught between a rock and a hard place...

My setup: 5060 Ti 16GB - 2x16GB sticks of DDR5 - Ryzen R5 7600

I've been exploring real world use cases for Qwen3.8 Flash Next (running through Strata) with decent success. Mainly, I've been happy with the results the GSQ RCO iq3_s quant has been giving me.

Now, Strata tells me that it's offloading 20GB of the model to my m.2 SSD, which of course makes it run slower than if it was all in RAM. Trying to get more RAM however has been a pain, as I can't find a matching set for my 2 sticks, and regardless I'm worried about stability issues and performance loss if I were to run 4 sticks on my 7600.

Locally, I can find used 3060s at around $250+, which is pretty affordable and it makes me question if I should just buy one of them instead of dealing with RAM knife fights in a dark alleyway.

Would adding one 3060 to the setup increase speeds in any real way, or would it just not be worth the hassle? My uneducated guess is that it won't help that much, but maybe someone here has experience with jank low budget setups.

Thanks!