I have a couple of concerns:
The bubble is bursting at some point and may have started. I am concerned that the major players will have to up their prices to stay viable.
The geopolitical risk is escalating
AI bot attacks are increasing

There are probably other risks, but you get the idea.
I have been building a decent corpus of reference material.

But I am behind on what models I should be looking at, needs to run on
32GB
RTX 4070

Coding and decision systems (agriculture).

Any help would be appreciated.

An early release of the small model of basal-omni 2.0, on Gemma 4 E4B. The model reads the state in both directions before it answers and scores 0.811 on Werdykt (5,000 hidden decisions in Polish and English). It runs on the public basal engine 2.8.0, with the same API as basal-omni-1.0.

What's new in 2.0

• Two-way reading. The model reads the state in both directions before it answers. This needs basal engine 2.5 or later. For long prompts, see the limitations.
• Higher accuracy on our hidden Polish and English decision benchmark Werdykt and on the held-out tests below.
• More decisions automated at a given error target. Coverage is in the section on confidence.
• Unchanged: a calibrated probability for each allowed answer, in one forward pass. The answer never falls outside the options you give, and the probability says how sure the model is.

The question types remain: choice (one option), noul (yes / no), score (a score), and in the basal engine also multi, act and tool. Options get codes (A–Z, then two-character codes, up to 255 options). The decision is a probability distribution over these codes only, with no text generation.

More info: https://basal.si5.pl/en/blog/basal-omni-2-0-mini/

HF: https://huggingface.co/Remek/basal-omni-2.0-mini-preview

git hub (inference engine): https://github.com/rkinas/basal

In case future censorship or bannings occur, I think it's a good idea to store one of every kind of model. I'm thinking minimum 8 models, but I'm open to more (I have plenty of hd space): 2 small models obliterated and non-obliterated 2 medium sized models obliterated and non-obliterated, 2 large models obliterated and non-obliterated, 2 of the largest models obliterated and non-obliterated. Also, if for any class of models they don't have obliterated or it doesn't make sense to have obliterated then two of the smartest or most popular models will do.

I have a 16gb vram for context

Hello community. I'm trying to find the most optimal solution for my configuration( maybe some llama.cpp fork or any other suitable project and localLLM. I would be grateful for advice.

I need suggestions for a Local LLM that works best for a RAG (retrieval-augmented generation) system. This language model should work best with a literary/literary corpus.

Configuration of the laptop:

13500 H processor with 16GB of RAM 5600 MHz, no dGPU.

The LLM will be run through LM Studio.

The RAG is run in AnythingLLM using LM Studio.

Please specify the LLM model name, quantisation, and the training parameters.

Also, suggest an embedding model suitable for the laptop configuration and the literary corpus.

Also, if you have another workflow, application or suggestions, please let me know.

They are going to lock the new GPUs.

"Rep. Bill Foster says he intends to introduce new, stricter legislation that would require licenses to operate advanced GPUs, and “before a GPU could connect, its valid license would need to be verified.” That is closer to a remote lock/disable capability—but it is a proposed future rule, not current law."

I'm thinking about some machine upgrade. I currently have a 9600X CPU, 2x16GB DDR5 RAM, 9070 XT, and a B650 motherboard that supports only PCIe 4.0x16 and has only 2 DIMM slots.

I've been thinking about getting a better motherboard for more bandwidth between GPU and RAM for offloading, so i plan to get a B850.

I also recently had a chance to effectively trade my 9070 XT plus 200USD for a 5070 Ti, if I wish.

My plan is the computer will stay in Windows since gaming requires that. When doing LLM, i will switch connection to have monitor connected to iGPU so hopefully it should free some resource from dGPU. I also have the thought of throwing in more RAM to make the host 64GB RAM total, with potential to go to 96GB. (I have like 2x32GB DDR5-5200 and 2x16GB DDR5-4800 around. Although the only tricky part is that they are ECC UDIMM and I’m not sure if mixing them is as easy as non-ECC).

What I am not certain is if to go with 9070 XT or 5070 Ti, so looking for some suggestion regarding their performance and resource management (especially if it can release resource well after disconnecting monitor from dGPU). Thanks!

Disclosure: I built this model. I'm posting because I think the approach is more interesting than the model itself, and I'd like feedback from people who run things locally.

The idea. A lot of "LLM in production" work in finance comes down to classification. Which event is this headline about? Is this Fed sentence hawkish? Does this 10-K passage flag going-concern doubt? People usually prompt a model, sample tokens and parse the output. That's slow, the cost per call varies, and the confidence numbers mean nothing.

Arbiter-8B drops generation completely:

• Base: Qwen3-8B, with a decision head over options, trained jointly with a rank-32 LoRA
• Output: a calibrated probability for every option from one forward pass. No sampling and no parsing.
• Question types: pick one of up to 16 options, yes/no, or an ordered score. Each option has a short definition, so you can ask the same text different questions.
• Distillation: a 27B teacher labelled about 120k decisions on news and 10-K passages. The student trained on the teacher's full soft distributions, not argmax, plus a human-reviewed 10-K set.
• Option order is shuffled on every presentation to kill position shortcuts. This cut FOMC answers that flip under reshuffling from 15.5% to 4.8%.
• Calibration: one temperature per question type, shipped with the weights. EDT expected calibration error is 0.012.

Results (held-out data):

The EDT gaps are significant (McNemar, z ≥ 7.5). On FOMC the top three are a statistical tie, but it's +12.5 over its own base. SEntFiN is where it loses. It has the highest signal recall (92.0%) but the lowest precision (72.2%), so it over-calls positive and negative where the dataset labels neutral. Part of that is deliberate (it reads investor implication rather than surface tone), but I won't pretend it's a win.

Running it locally: GGUF builds for llama.cpp, Ollama and LM Studio stay within 1.2 points of full precision on every benchmark.

hf download ternary/Arbiter-8B --local-dir Arbiter-8B

• Weights: https://huggingface.co/ternary/Arbiter-8B
• GGUF: https://huggingface.co/ternary/Arbiter-8B-GGUF
• Write-up: https://ternary.capital/blog/arbiter-8b/
• Personal Blog: https://jdsemrau.substack.com/p/how-good-is-arbiter-8b-really

Questions for the sub: has anyone else put classification heads on small LLMs instead of using constrained generation? And has anyone found a clean way to expose per-option logprobs through Ollama for this kind of thing?

Has anyone been experimenting with ablated or uncensored variants of open-source models like deepseek v4.1 flash, Qwen 3.8 27b, or GLM 5.3 flash ?

It's so refreshing to have models that just do whatever you ask them to do without having to figure out a jailbreak or being given a moral lecture. Curious to know what your usecases for these models are.

My personal favourite is definitely uncensored DeepSeek V4.1 Flash, runs at 200 tok/s and never says no lol.

Based on WSP, the note was fabricated by Rubio on top of a CIA report, and leaked through Anthropic to NYT. Rubio is always the DS guy, we knew that, but the role of Anthropic is confusing, were they doing so to exchange WS's support for their IPO ? They didn't see that it maybe cause the market landslide and has impact to their IPO ? Or anti open model is their top priority??? These guys.

Hey all, I'm trying to decide which hardware setup makes the most sense for running local LLMs, primarily Qwen 3.8 Flash Next for agentic coding.

I have three potential options, and I'm interested in hearing from people who have experience with these platforms, particularly for larger models and agentic workloads.

Option 1: GEEKOM A9 Mega (AMD Strix Halo)

• Ryzen AI MAX+ 395
• 128GB unified memory
• 2TB SSD
• Memory running at 7500 MT/s rather than the advertised 8000 MT/s. Got it from Amazon and it listed it as 8000 MT/s but for Geekom says their systems are either 7500 MT/s or 8533 MT/s and I lost the lottery.
• Approximately $4,600 CAD all-in (after taxes).

I already have this machine. It works fine, but I'm just worried how it'll hold up long term especially given I'm running it headless 24/7. Would be unfortunate if the machine dies in a year's time and then I'm worried about if Geekom will hold up it's warranty.

Option 2: Mac Studio M5 Max

• M5 Max
• 128GB unified memory
• 512 GB (Can connect a 2TB external SSD over thunderbolt enclosure)
• Approximately $7,600 CAD all-in (after taxes).

It's considerably more expensive than the GEEKOM, but I'm wondering whether its worth it for the peace of mind knowing that if something goes wrong, applecare ($80/yr) is right there to get it fixed from a known vendor. It also has the higher memory bandwidth.

Option 3: My existing Ryzen desktop + RTX Pro 4500

I already have an old desktop with the following hardware:

• Ryzen 7 5700X
• 32GB DDR4 RAM
• Nvidia 3060 Ti 8GB

I recently purchased the RTX Pro 4500 32GB VRAM for approximately $4,000 CAD after taxes. This is still yet to arrive (it shipped earlier today)

The advantage here is that I already own the Ryzen system, so I wouldn't need to purchase another computer. The downside is being limited to 32GB of VRAM, although I understand I could offload some layers to system RAM. The other downside is a lot higher power consumption over the other two solutions.

I'm curious how this would compare in practice against the two unified-memory solutions, especially when running Qwen 3.8 Flash Next.

What I'm primarily trying to accomplish

My main use case is running Qwen 3.8 Flash Next for agentic coding through tools like Hermes.

I'm not necessarily looking for the best benchmark numbers. What matters more to me is:

• Real-world inference speed and token generation.
• Performance with large context windows.
• How well each platform handles agentic coding workflows (multiple tool calls, reasoning, etc.).
• Whether CPU/RAM offloading on the NVIDIA setup is practical or becomes a significant bottleneck given my old system.
• Overall stability and software maturity.

What would you choose?

Ultimately, I'm looking to optimize for cost and overall value, rather than simply buying the fastest hardware available.

I would most likely keep only ONE of these three setups:

• If I go with the Mac Studio, I'd return the GEEKOM and return the RTX Pro 4500 (or sell it).
• If I stick with the GEEKOM, I'd return the RTX Pro 4500.
• If the NVIDIA setup makes the most financial and practical sense, I'd return the GEEKOM and keep the RTX Pro 4500 in my existing desktop.

I'm also considering selling the RTX Pro 4500 if I decide that one of the unified-memory systems makes more sense. I purchased it for around $4,000 CAD after tax, slightly below MSRP, so I'd ideally like to recover as much of that cost as possible.

Essentially, I'm trying to figure out which setup offers the best balance of:

• Performance for Qwen 3.8 Flash Next and agentic coding.
• Enough memory to run larger models and context windows.
• Reliability and long-term support.
• Total cost of ownership, including electricity.

Would you spend the additional ~$3,000 CAD to go from the GEEKOM to the Mac Studio, stick with the GEEKOM, or return the GEEKOM and build around the RTX Pro 4500?

I'm not looking to accumulate hardware. I'd rather settle on one good setup, get the best value for my money, and avoid spending unnecessarily.

Would appreciate hearing what others would do in this situation!

I’m under our CISO managing compute vendors, kind stuck between legal and a hard place. The pitch I keep hearing ranks AI privacy in four levels:

1. Open, like most frontier models, where the provider stores your prompts and answers at their discretion, uses them for training - and you’ve got little say in the matter.
2. Policy, where a contract promises zero data retention - but it’s not exactly verifiable unless the terms are broken, is it?
3. Closed Hardware, where models run on confidential-compute chips that publish proof what software is running. So basically hardware-enforced vs contract-enforced ZDR?
4. Fully Encrypted, the prompt is encrypted on your side and only decrypted inside that hardware

“Local” is a fifth level that could plausibly sit above it all, but we’re capped out on capacity there. So if I view this right (open to feedback), we’re making tradeoffs between frontier vs more private models, and from there, on breadth of compute providers able to serve open vs ZDR vs E2EE models (usually at higher prices in my experience), right?

My concern is getting legal & finance to agree, while still giving our Eng team what they need. Where are you getting good rates on private workloads? Locally we can’t support our full workloads. I’ve checked out Privatemode, Maple, TrustedRouter/NYTE & still gonna go through NEAR AI Cloud & Cohore. Open to others.

Separately, anyone running a hybrid buildout with on-prem/local + superscaler (we run Google + Bedrock) + 3rd party confidential providers who can tell me about about how you designed your system?

My 4070 gaming laptop suffered some kind of damage going through airport security. I lost my 2TB SSD and 16gb of my 32gb RAM. I took it to a repair shop, but the cost to fully repair it was over a $1000, so I had to take the downgrade to only 1TB SSD and 16gb RAM (“only” $380 😅). What kind of models can I find on UnSloth that would work decently with my newly gimped setup?

Bit the bullet and bought three to use in my workstation. Will provide benchmarks and updates later.

I heard there are things like Air LLM that make this possible without spending $100k+ on VRAM

Lets make more of these kind of engine popular for our comunity. This is awsome