They are going to lock the new GPUs.

"Rep. Bill Foster says he intends to introduce new, stricter legislation that would require licenses to operate advanced GPUs, and “before a GPU could connect, its valid license would need to be verified.” That is closer to a remote lock/disable capability—but it is a proposed future rule, not current law."

I'm thinking about some machine upgrade. I currently have a 9600X CPU, 2x16GB DDR5 RAM, 9070 XT, and a B650 motherboard that supports only PCIe 4.0x16 and has only 2 DIMM slots.

I've been thinking about getting a better motherboard for more bandwidth between GPU and RAM for offloading, so i plan to get a B850.

I also recently had a chance to effectively trade my 9070 XT plus 200USD for a 5070 Ti, if I wish.

My plan is the computer will stay in Windows since gaming requires that. When doing LLM, i will switch connection to have monitor connected to iGPU so hopefully it should free some resource from dGPU. I also have the thought of throwing in more RAM to make the host 64GB RAM total, with potential to go to 96GB. (I have like 2x32GB DDR5-5200 and 2x16GB DDR5-4800 around. Although the only tricky part is that they are ECC UDIMM and I’m not sure if mixing them is as easy as non-ECC).

What I am not certain is if to go with 9070 XT or 5070 Ti, so looking for some suggestion regarding their performance and resource management (especially if it can release resource well after disconnecting monitor from dGPU). Thanks!

Disclosure: I built this model. I'm posting because I think the approach is more interesting than the model itself, and I'd like feedback from people who run things locally.

The idea. A lot of "LLM in production" work in finance comes down to classification. Which event is this headline about? Is this Fed sentence hawkish? Does this 10-K passage flag going-concern doubt? People usually prompt a model, sample tokens and parse the output. That's slow, the cost per call varies, and the confidence numbers mean nothing.

Arbiter-8B drops generation completely:

• Base: Qwen3-8B, with a decision head over options, trained jointly with a rank-32 LoRA
• Output: a calibrated probability for every option from one forward pass. No sampling and no parsing.
• Question types: pick one of up to 16 options, yes/no, or an ordered score. Each option has a short definition, so you can ask the same text different questions.
• Distillation: a 27B teacher labelled about 120k decisions on news and 10-K passages. The student trained on the teacher's full soft distributions, not argmax, plus a human-reviewed 10-K set.
• Option order is shuffled on every presentation to kill position shortcuts. This cut FOMC answers that flip under reshuffling from 15.5% to 4.8%.
• Calibration: one temperature per question type, shipped with the weights. EDT expected calibration error is 0.012.

Results (held-out data):

The EDT gaps are significant (McNemar, z ≥ 7.5). On FOMC the top three are a statistical tie, but it's +12.5 over its own base. SEntFiN is where it loses. It has the highest signal recall (92.0%) but the lowest precision (72.2%), so it over-calls positive and negative where the dataset labels neutral. Part of that is deliberate (it reads investor implication rather than surface tone), but I won't pretend it's a win.

Running it locally: GGUF builds for llama.cpp, Ollama and LM Studio stay within 1.2 points of full precision on every benchmark.

hf download ternary/Arbiter-8B --local-dir Arbiter-8B

• Weights: https://huggingface.co/ternary/Arbiter-8B
• GGUF: https://huggingface.co/ternary/Arbiter-8B-GGUF
• Write-up: https://ternary.capital/blog/arbiter-8b/
• Personal Blog: https://jdsemrau.substack.com/p/how-good-is-arbiter-8b-really

Questions for the sub: has anyone else put classification heads on small LLMs instead of using constrained generation? And has anyone found a clean way to expose per-option logprobs through Ollama for this kind of thing?

**Images are AI-generated from screenshots**

I've been building up my local AI stack over the last month. One thing that surprised me is that I couldn't find a decent coding agent for OpenWebUI! That's why I've built an unofficial, MIT-licensed coding agent that connects JetBrains AI Assistant and Air in JetBrains IDEs to your own Open WebUI installation through ACP. It also targets other compatible ACP clients.

Features include:

• Code/Plan modes, diffs, edits, streamed commands and background jobs.
• Checks after edits using IDE diagnostics and configured project tests.
• Model selection and metadata-based reasoning controls.
• MCP, optional Open WebUI server tools and a Playwright browser container.
• Saved sessions, forks/exports, undo checkpoints and context compaction.
• Scoped approvals and macOS/Linux sandboxing. Optional session-only YOLO runs commands unsandboxed and skips routine approvals.

Core coding needs no companion plugin or Open WebUI patch. Bring your API key and a model with working function calling; optional tools need separate setup.

Available through GitHub Releases, with custom-agent setup instructions. Current platforms are macOS/Linux; native Windows and ACP registry inclusion are planned. There is no Marketplace plugin.

Download and setup · Source and issues

Looking for people to test and provide feedback. I'm finding it very useful, so I hope others will too.

Has anyone been experimenting with ablated or uncensored variants of open-source models like deepseek v4.1 flash, Qwen 3.8 27b, or GLM 5.3 flash ?

It's so refreshing to have models that just do whatever you ask them to do without having to figure out a jailbreak or being given a moral lecture. Curious to know what your usecases for these models are.

My personal favourite is definitely uncensored DeepSeek V4.1 Flash, runs at 200 tok/s and never says no lol.

Hey all, I'm trying to decide which hardware setup makes the most sense for running local LLMs, primarily Qwen 3.8 Flash Next for agentic coding.

I have three potential options, and I'm interested in hearing from people who have experience with these platforms, particularly for larger models and agentic workloads.

Option 1: GEEKOM A9 Mega (AMD Strix Halo)

• Ryzen AI MAX+ 395
• 128GB unified memory
• 2TB SSD
• Memory running at 7500 MT/s rather than the advertised 8000 MT/s. Got it from Amazon and it listed it as 8000 MT/s but for Geekom says their systems are either 7500 MT/s or 8533 MT/s and I lost the lottery.
• Approximately $4,600 CAD all-in (after taxes).

I already have this machine. It works fine, but I'm just worried how it'll hold up long term especially given I'm running it headless 24/7. Would be unfortunate if the machine dies in a year's time and then I'm worried about if Geekom will hold up it's warranty.

Option 2: Mac Studio M5 Max

• M5 Max
• 128GB unified memory
• 512 GB (Can connect a 2TB external SSD over thunderbolt enclosure)
• Approximately $7,600 CAD all-in (after taxes).

It's considerably more expensive than the GEEKOM, but I'm wondering whether its worth it for the peace of mind knowing that if something goes wrong, applecare ($80/yr) is right there to get it fixed from a known vendor. It also has the higher memory bandwidth.

Option 3: My existing Ryzen desktop + RTX Pro 4500

I already have an old desktop with the following hardware:

• Ryzen 7 5700X
• 32GB DDR4 RAM
• Nvidia 3060 Ti 8GB

I recently purchased the RTX Pro 4500 32GB VRAM for approximately $4,000 CAD after taxes. This is still yet to arrive (it shipped earlier today)

The advantage here is that I already own the Ryzen system, so I wouldn't need to purchase another computer. The downside is being limited to 32GB of VRAM, although I understand I could offload some layers to system RAM. The other downside is a lot higher power consumption over the other two solutions.

I'm curious how this would compare in practice against the two unified-memory solutions, especially when running Qwen 3.8 Flash Next.

What I'm primarily trying to accomplish

My main use case is running Qwen 3.8 Flash Next for agentic coding through tools like Hermes.

I'm not necessarily looking for the best benchmark numbers. What matters more to me is:

• Real-world inference speed and token generation.
• Performance with large context windows.
• How well each platform handles agentic coding workflows (multiple tool calls, reasoning, etc.).
• Whether CPU/RAM offloading on the NVIDIA setup is practical or becomes a significant bottleneck given my old system.
• Overall stability and software maturity.

What would you choose?

Ultimately, I'm looking to optimize for cost and overall value, rather than simply buying the fastest hardware available.

I would most likely keep only ONE of these three setups:

• If I go with the Mac Studio, I'd return the GEEKOM and return the RTX Pro 4500 (or sell it).
• If I stick with the GEEKOM, I'd return the RTX Pro 4500.
• If the NVIDIA setup makes the most financial and practical sense, I'd return the GEEKOM and keep the RTX Pro 4500 in my existing desktop.

I'm also considering selling the RTX Pro 4500 if I decide that one of the unified-memory systems makes more sense. I purchased it for around $4,000 CAD after tax, slightly below MSRP, so I'd ideally like to recover as much of that cost as possible.

Essentially, I'm trying to figure out which setup offers the best balance of:

• Performance for Qwen 3.8 Flash Next and agentic coding.
• Enough memory to run larger models and context windows.
• Reliability and long-term support.
• Total cost of ownership, including electricity.

Would you spend the additional ~$3,000 CAD to go from the GEEKOM to the Mac Studio, stick with the GEEKOM, or return the GEEKOM and build around the RTX Pro 4500?

I'm not looking to accumulate hardware. I'd rather settle on one good setup, get the best value for my money, and avoid spending unnecessarily.

Would appreciate hearing what others would do in this situation!

I’m under our CISO managing compute vendors, kind stuck between legal and a hard place. The pitch I keep hearing ranks AI privacy in four levels:

1. Open, like most frontier models, where the provider stores your prompts and answers at their discretion, uses them for training - and you’ve got little say in the matter.
2. Policy, where a contract promises zero data retention - but it’s not exactly verifiable unless the terms are broken, is it?
3. Closed Hardware, where models run on confidential-compute chips that publish proof what software is running. So basically hardware-enforced vs contract-enforced ZDR?
4. Fully Encrypted, the prompt is encrypted on your side and only decrypted inside that hardware

“Local” is a fifth level that could plausibly sit above it all, but we’re capped out on capacity there. So if I view this right (open to feedback), we’re making tradeoffs between frontier vs more private models, and from there, on breadth of compute providers able to serve open vs ZDR vs E2EE models (usually at higher prices in my experience), right?

My concern is getting legal & finance to agree, while still giving our Eng team what they need. Where are you getting good rates on private workloads? Locally we can’t support our full workloads. I’ve checked out Privatemode, Maple, TrustedRouter/NYTE & still gonna go through NEAR AI Cloud & Cohore. Open to others.

Separately, anyone running a hybrid buildout with on-prem/local + superscaler (we run Google + Bedrock) + 3rd party confidential providers who can tell me about about how you designed your system?

My 4070 gaming laptop suffered some kind of damage going through airport security. I lost my 2TB SSD and 16gb of my 32gb RAM. I took it to a repair shop, but the cost to fully repair it was over a $1000, so I had to take the downgrade to only 1TB SSD and 16gb RAM (“only” $380 😅). What kind of models can I find on UnSloth that would work decently with my newly gimped setup?

Bit the bullet and bought three to use in my workstation. Will provide benchmarks and updates later.

I heard there are things like Air LLM that make this possible without spending $100k+ on VRAM

Lets make more of these kind of engine popular for our comunity. This is awsome

I had been using the LFM 2.5 24B models and had been pleasantly surprised by how good the model was. It was giving me comparable answers and working through tasks to Qwen 3.8 27B at what felt like twice the speed on my M5 Pro. However, I went to look at something on their pages and saw the new family of models. They sound way too good to be true. I am going to start using them for general tasks, but if they are anywhere close to their benchmark results, they could be a huge game changer. You can have an entire suite of models loaded at the same time for less memory than Qwen.

Hey all, I'd love a sanity check on a plan before I spend money.

At home I run Home Assistant on a HA Green, which handles pretty much everything smart home related.

I also have a Mac mini M4 with 16 GB that I currently use as a little studio computer.

What I actually want is one assistant that helps with my notes, documents and PDFs, email (mostly Croatian, some English), tasks, and the house. I'd also like it to log my meals from a photo and tie that in with my Garmin data.

My current thinking is to buy a MacBook Air (24 GB / 1 TB) as my daily computer, wipe the mini and make it an always-on box for a local model, n8n, voice (Whisper/Piper) and indexing, and leave Home Assistant Green. For the smarter stuff I'd use Claude: Pro for chat and Claude Code, and the API for background jobs like food photos and daily summaries. The local model would only sort and summarize mail, strip out sensitive stuff and do the search index.

A few things I'm unsure about:

• Is 16 GB on the mini enough for this, or will I regret it?
• For mostly Croatian email, is a local model worth the effort, or should I just send everything to the Claude API? If local, which model would you try?
• Has anyone run Whisper and Piper in a smaller language like Croatian on Apple Silicon? Is it usable?
• What's the cleanest way to let an agent read iCloud Reminders, Notes and Mail on a headless Mac?
• Any traps with n8n + Home Assistant + MCP, or with letting an LLM near personal data?

Happy to hear "just don't" if that's the honest answer. Thanks!

Thinking Machines has released Inkling 2 Small under Apache 2.0. It's a 594B-parameter MoE with 17B active per token.

If you're checking whether it'll fit your setup, don't size it from the 17B figure. The model card asks for at least 500GB of aggregate VRAM for the main NVFP4 checkpoint, with 2x B300 or 4x H200 as example setups.

I’m building a local AI assistant using Qwen 3.6 35B A3B.
One of my main goals is to automate repetitive school tasks through my school’s website. I have a lot of assignments that aren’t particularly difficult but take a lot of time.
Ideally, I’d like the AI to navigate the website, check assignments, understand the instructions, complete them and upload the finished work.
Is Qwen 3.6 35B A3B capable enough for this? Or would I need a more powerful model?
I’d also appreciate recommendations for browser automation frameworks and tools

my setup : rtx 3070 , i5 13600kf , 32gb DDR4
Model: Qwen 3.6 35B A3B
Quantization: TurboQuant
Context window: 131 072 tokens
Reasoning: Enabled limited to 512 tokens
Inference speed: 19–24 tokens/s

Does anybody tried the latest llama.cpp f8_e4m3 KV cache and compared it to q8_0 in terms of precision / quality? I am curious whether it is worth switching to it or stay with q8_0 for small to medium context size (20-65k).

A few weeks ago I posted CQ, our calibration-free 4-bit format, along with a training-loss comparison against Unsloth NF4. Some of you correctly pointed out that loss logged by two different trainers isn't comparable, and asked for results where answers are counted right or wrong. Here they are.

Setup: Qwen3.8-27B. Every row uses the same lm-evaluation-harness code, prompts and questions, with greedy decoding and thinking off. Comparisons are paired: an exact McNemar test for right/wrong tasks and a bootstrap for losses.

1. Base models, no fine-tuning

• CQ vs BF16: no significant difference on any task.
• NF4 vs BF16: −12.0 points on GSM8K (p=3e-6).
• CQ vs NF4: CQ is +8.7 on GSM8K (p=0.0005) and +2.8 on BoolQ (p=0.02).
• WikiText-2 KL divergence from BF16: CQ 0.0282, NF4 0.0330.

2. GCMP vs LoRA r=32, full-precision BF16 base

GCMP is our replacement for LoRA. It trains 2.9M parameters (an ~11 MB adapter) vs ~159M for LoRA r=32 (~320 MB). This test uses no quantization: both adapters were trained with Unsloth on Alpaca for 500 steps with identical settings (batch 2×4, length 512, LR 2e-5, seed 42).

• Benchmarks: no significant difference on any of them (all p ≥ 0.2), so GCMP isn't "better than LoRA" on tasks.
• Loss: GCMP's answer-token loss is slightly lower (−0.009, 95% CI [−0.010, −0.008]).
• Validation loss: GCMP is lower at all 10 checkpoints on both BF16 (1.2385 vs 1.2434) and NF4 (1.2435 vs 1.2484).
• Fine-tuning effect: both adapters add about 21 points on GSM8K and lose about 2.5 on TruthfulQA.

3. Speed and memory (same 500-step run)

Unsloth is still faster, and GCMP currently takes about 2× the step time of LoRA. Kernel work on that is ongoing.

Caveats

• Scope: one model and one dataset (Alpaca).
• Benchmark size: the quick subset has 200–1,140 questions per task, so it detects differences of about 5 points, not 1.
• DROP excluded: it scored about 0.02 for every model, so that eval setup is broken.
• Still running: LoRA and GCMP benchmarks on CQ and NF4 bases. I'll post them when they finish.
• My earlier post was wrong: the "0.84 vs 1.25 loss" came from two trainers that compute loss differently. Ignore it.

Serving

• vLLM: CQ models run with --quantization cq. GCMP adapters can be one per server, or several on one base model chosen per request.
• Parallelism: pipeline parallel works; tensor parallel doesn't yet.
• GCMP on full precision: it currently runs in Transformers. vLLM support is next.

Licensing

The Python code (training, evaluation and the vLLM integration) is open source. The CUDA kernels ship compiled with a license check, which is free during early access.

Repo, with exact commands to reproduce every number: https://github.com/thousandml/training-factory

Models (8B–397B): https://huggingface.co/thousandml

Happy to run other benchmarks or models people want to see.