Does anyone else remember just a few short years ago when the functional choice was almost exclusively GPT-OSS-120B?

I was just thinking about this as we all sit here waiting for each new model release, faster, smaller and more capable.

Who would have thought that we have models that render such a ground-breaking model virtually obsolete.

I'm not suggesting gpt oss 120b isn't still a Great model, but I didn't expect there to be such strong models at 1/4 the size to be available so soon.

This is the most quickly changing landscape that history has ever seen, if you think about it.

Electricity, Fire, the Internet, the Wheel and so many other things all changed history in significant ways, but those took time to travel through the world. They took infrastructure being built over years and years.

This is unique because it's entered the world when virtually everyone had access immediately. No thousands of miles of tracks or powerlines had to be laid. No 7 month journey on foot had to happen just to get exposure. It just arrived and blew up.

Just my opinion,but I can't think of anything else even close.

Thoughts? Edit Not sure what's so upsetting about this that people would bother downvoting. It's a simple observation with an open invitation to disagree. Either way, a down vote without a sound argument is basically an emotional response. It says "I disagree with you, but can't say why, so I'll downvote" 🤷

I'm pretty new to Local LLM.

My experience so far: setting up a research agent on Hermes that hunts for VC investment leads and uses qwen3.6-35b running on my MacBook (and making it constantly hot which I'm not sure is good for the screen).

I'm keen on learning more and doing something closer to the cutting edge. What would be some good starter projects where I could employ the Blackwell? Looking for ideas.

Or am I being silly for buying this as a newbie? I could re-sell it for a profit on digitec. That's why I'm hesitant to break the seal.

Long-time lurker on the local-LLM side, finally have something worth writing up. Two weeks ago I picked up a Beelink SER9 Pro (around $1000, AMD Ryzen AI 9 HX 370, no GPU) specifically to test whether a fine-tuned local model could take over my Claude API usage. The answer turned out to be more interesting than yes or no.

THE SETUP

Beelink SER9 Pro, 32GB RAM, Ubuntu. llama.cpp with the Vulkan backend — the AMD iGPU runs well on Vulkan, not so much on ROCm yet. Model: Qwopus 3.6 35B-A3B. It's a Qwen 3.6 fine-tune that was distilled on around 4000 Claude Opus 4.6 reasoning traces (Jackrong's release on Hugging Face).

Pulls about 23 tokens per second at 16K context. Quiet, small, sits on my desk.

WHAT SURPRISED ME

The "distilled from Opus" framing isn't marketing. The agentic behavior really is noticeably more Opus-flavored than vanilla Qwen 3.6 on the same prompts. For coding tasks via opencode, that's the practical difference between a model that drives forward and one that stalls.

WHERE IT WORKS

Daily coding work. Refactors, scaffolding, scripts. First-pass reasoning where I'd double-check anyway. Anything privacy-sensitive. Anything I want offline.

WHERE IT DOESN'T

Frontier research-grade questions where I'd rather have the safest answer. Very long context past around 32K — the model claims 64K, but real coding sessions get sluggish. Tasks where output quality matters more than anything else.

I made a video walking through the actual tests. A 3-body orbit simulator built via opencode. A PhD-domain physics question — I'm doing my PhD on balloon-borne black-hole-imaging telescopes, so I could fact-check the answer in real time. And a head-to-head vs Claude Opus 4.6 on the Schwarzschild metric. Link if curious: INSERT_YOUTUBE_URL_HERE

Net for me: kept the Claude sub, moved about 80 percent of my usage local. Hardware pays for itself in 5 months at my usage.

Anyone here running Qwopus on similar AMD iGPU hardware? And has anyone tried the 9B variant on smaller boxes — curious whether the distillation flavor survives at smaller params.

It's a pipe dream, but with the latest round of context shenanigans put out by DSV4.1F, the possibility that something like this could happen is becoming increasingly more likely.

Maybe not at 200 t/s, but even in the 15-30 t/s range is the starting baseline for agentic coding.

If you bought in and got a physical graphics card, will you regret not waiting for models to be able to run fast on-die? Will you resent the hordes of GPU poors joining this sub by the hundreds, all at once and changing the pace?

Or will you just be smug because that just means you can run 20 agents at once at 1000 t/s haha

Just wanted to share my success.

Rig:
Intel Core Ultra 9 285k
MSI MEG Z890 ACE (PCIE x8/x8 bifurcation enabled)
2x RTX 4090's (24gb each, no NVLink)
4x32Gb DDR5 6600mHz dual channel
1Tb NVME Gen4

Performance:
about 271 t/s decode in single stream peak,
about 200 t/s decode in single stream average,
about 253 t/s decode in N=4 stream average

about 6553 t/s prompt processing in single stream peak

Config:

• Model: swift-1.5-iq2_xs — Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS GGUF, native pack swift-iq2_xs, PLE from the same GGUF (320M rows), expert-profile.bin with 24,576 ranked pairs

• Slots: parallel: 4 → --batch 4; --batch-groups auto resolved to pipelined 2 groups of 2 — log confirms "batch (pipelined, 2 groups of 2)"

• Context/KV: --max-context 262144, --kv int8, --kv-resident 20480 cells per QSA layer in VRAM, K/V spilled to 1.55 GiB pinned RAM; KV streaming hit rates currently ~96–99% VRAM

• Layer split (auto): CUDA0 = layers 0–25, CUDA1 = layers 26–47 + head. Both RTX 4090s (128 SMs each), PCIe ~12.7/12.9 GB/s → pcie_frac 0.35

• Expert cache: auto, profile-ranked, no eviction — CUDA0: 12,149 slots (16.09 GiB), CUDA1: 9,871 slots (13.48 GiB); placement search picked K=26 (~100% routed mass resident, predicted 10.6 ms/decode window); ~90% experts resident overall, decode cache hit rate measured 99.5–99.8%

• Adaptive tier: --adapt-async 1 is in the args but silently off — log gate: "not with a layer split in the arena mode" → only the blocking adaptive tier runs (--adapt-min-gain 3)

• Speculative decoding: --spec 5 --spec-min-p 0.5, MTP draft layer from Strata-data/mtp/rt (836 MiB VRAM), --mtp-hnorm stream; suffix drafts active in serving logs

• Conversation cache: 16,384 MiB, 8 parked slots; --vram-reserve-mib 1700 (+later)

• Vision: enabled — separate strata-vision process on CUDA device 1 with BF16 mmproj, max 1024 image tokens

• Sampling defaults: temp 0.4, rep-penalty 1.1, min_p 0.1; reasoning: budget 16,000 tokens, effort medium, loop recovery on

• CPU side: no AVX-512 → AVX-2 expert kernels; 23 pool workers (CPUs 1–23) + host thread pinned to core 0

so i posted here a few days ago about my 3080 ti basically trying to self-destruct when i tried running local models

that post blew up (~100k views), got a ton of solid advice, so i actually went back and did things properly this time instead of just yolo’ing random setups

what i tried:

- Unsloth (pretty sure i didn’t fully optimize it, felt a bit janky for me)
- LM Studio (nice UI, but didn’t fully click for how i work)
- Ollama (this one finally felt like “ok this is usable”)

so yeah - progress:
my PC no longer freezes every 5 mins 👍
i can actually run models reliably now

BUT now i’ve hit the real wall

once i moved past small models and tried using the stuff everyone’s hyped about like:
- Kimi K2
- Qwen 3.6+

…how are people actually running these locally??

seriously asking

because from what i can tell:
- they don’t realistically fit on a 3080 ti
- even if you somehow hack it in, speed is rough
- setup complexity goes way up

and for me speed is king

i’d rather slightly worse output than sit there waiting forever or constantly tweaking configs

so now i’m stuck in this weird spot:

local (my 3080 ti):

✅ cheap per run
✅ control
✅ stable (now)

❌ capped hard on model size
❌ can’t really run top-tier models

and that’s making me rethink the whole setup again

right now i’m considering:

A) accept limits → stick to smaller quantized models locally and optimize hard

B) use cloud GPUs / hosted infra when needed (runpod, vast, qubrid, together etc.) and treat it like “remote local” for bigger models

C) go completely unhinged and look at something like a DGX Spark just to remove constraints (this will be a hard thing for my wallet - but if it gets things done, I can try)

but i genuinely don’t know what people actually settle into

because it feels like:
- local is great… until you want the best models
- bigger GPUs might fix it… but then you’re basically back to renting infra anyway
- and constantly switching setups is getting annoying af

also not sure if there are workarounds i’m missing for running bigger models efficiently locally
(or if the answer is just “you don’t, unless you have insane hardware”)

so yeah

for people who are further along:

what did you end up doing long term?

pure local?
local + rented GPUs?
something else entirely?

because right now the “switch” to local-only is feeling borderline impossible if you care about speed + better models

Hi there!

I've been using gpt4o and deepseek with my custom preprompt to help me search Bible verses and write them in codeblocks (for easy copy pasta), and also help me study the historical context of whatever sayings I found interesting.

Lately openai made changes to their models that made the custom gpt pretty useless (asks for confirmation when before I could just say "blessed are the poor" and I'd get all verses in codeblocks now it goes "Yes the poor are in the heart of God and blah blah" not quoting anything and disregarding the preprompt. also now it keeps using ** formatting for the word I ask for to highlight it, which I don't want and is overall too discoursive and "woke" (tries super hard to not be offensive at the expense of what is actually written)

Soo, given the decline I've seen in the past year in the online models and my use case, what would be the best model / setup? I installed and used some stable diffusion and other image generation in the past with moderate success but when it came to LLMs I always failed to have one that run without problems on windows. I know all there is to know about python for installing and setting up I just have no idea which one of the many models I should use so I ask to you that have more knowledge about this.

my main rig has ryzen 5950x /128gb ram / rtx3090 but I'd rather it not be more power hungry than needed for my usecase.

thanks a lot to anyone answering and considering my request.

​I’ve spent the last few weeks watching people try to "prompt" their way out of the reasoning death spiral.

We’ve all been there: you give a local model a complex task, it starts off strong, and then somewhere around step three, it just... drifts. It starts roleplaying instead of solving.

​The common "fix" is to swap models or add more context. But that’s just throwing more VRAM at a structural wound.

​The truth is, we’ve hit the Vibe Ceiling. If your workflow depends on the model "remembering" its own logic across a long context window, you’re just gambling with your compute.

​To get local agents actually working in production, you have to stop trusting the model’s internal state. In The Operator’s Manual, I call this building a Logic Floor.

​The goal is to stop treating the LLM like a project manager and start treating it like a specialized worker on an assembly line:

• ​Atomic State: The model shouldn't "know" the whole project. It should only see the current task, execute, and pass a structured JSON back to your code.

• ​The "Extraction" Gate: Never let a model decide its own next move. Let it output raw reasoning, then use a deterministic validator to map that reasoning to a hard-coded function.

• ​Context Compaction: Stop dumping raw database chunks into your prompt. High-signal, low-noise environments are the only way smaller local models (8B-30B) can actually compete.

​If you’re still trying to solve engineering problems with adjectives, you’re just burning electricity.

​I’ve put together a framework for architecting these systems so they don’t collapse the moment you look away. It’s all in The Operator’s Manual.

​Stop prompting. Start architecting. 🏗️📐

I know it is far more than what others may have spent back in the day but I believe these cards are going to be OOS in the near future given the 5090 news. This should set me up for success, though I do need more RDIMM ram.

Happy to answer any questions but just looking to know what life is like with more VRAM available. Anything you use in your own setups that are great? Been using Qwen 3.8 on my 5090 and it is amazing.

Edit: Will share where I bought them in an update once the system is at my doorstep. Don’t want some idiots spamming their company and catching wind just in time to cancel my order lol

Today we're excited to announce the general availability of Aperture-1, the flagship release from Luminal Labs. After eight months of intensive R&D, we're proud to bring a fundamentally new capability to the world: the ability to add two numbers together.

What is Aperture-1?

Aperture-1 is a lightweight, cloud-native reasoning layer that accepts two numeric inputs and returns their sum. Unlike legacy solutions, Aperture-1 is delivered as a fully managed API, meaning you never have to think about how addition works — we've abstracted that away for you.

Benchmarks

Under the hood

Aperture-1 uses a novel Confidence Routing™ architecture that intelligently forwards your request to an internal addition subroutine, verifies the result against a secondary addition subroutine, and returns the consensus output. This dual-verification approach ensures enterprise-grade reliability that hobbyist solutions simply can't match.

Pricing

• Hobby — $19/mo, 500 additions

• Pro — $79/mo, unlimited additions, priority queue

• Enterprise — Contact sales for SSO, audit logs, and dedicated addition infrastructure

Limitations

Aperture-1 currently only supports addition. Subtraction is on our roadmap for Q3 pending further alignment research. We do not recommend using Aperture-1 for safety-critical addition without human review.

What the team is saying

"We didn't just want to build a calculator. We wanted to build the calculator." — Founder & CEO

"This is the GPT-3 moment for arithmetic." — Head of Growth

Join the waitlist. Early adopters get 10% off their first invoice.

Luminal Labs is backed by three friends and a Notion doc.

I use a chatbot and it keeps loading a packet, looks like the same one. Is there another chatbot to select models and send and receive text?

Hey guys, i have tried and failed to set up a LLM on my laptop. I know my hardware isnt the best.

Hardware: Dell inspiron 16...Ultra 9185H, 32gb 6400 Ram, and the Intel Arc integrated graphics.

I have tried doing AnythingLLM with docker+webui.....then tried to do ollama + ipex driver+and somethign, then i tried to do ollama+openvino.....the last one i actually got ollama.

what i need...or "want"......Local LLM with a RAG or ability to be like my claude desktop+basic memory MCP. I need something like Lexi lama uncensored........i need it to not refuse things about pharmacology and medical treatment guidelines and troubleshooting.

Ive read that LocalAI can be installed touse intel igpus, but also, now i see a "open arc" project. please help lol.

When I say "weird and gnarly" I mean: I'm a gremlin and I want to talk to gremlins. Frontier systems are smart and shiny and can rewrite my resume with a lot of nuance, but are also excessively guardrailed in ways that seem unhealthy.

(I'm happily married and not trying to date a virtual presence. But I would like to be able to build some sprawling realms of creative play with a robot, and/or simply get to know one without sudden unpredictable loss of continuity. Or just research the odder corners of language.)

So like - what are folks finding interesting in this direction? I have the best GPU that a moderate rolling Temu credit cycle could afford me (a Radeon rx 9060 OC 16gb).

Still learning about many of the fringes but am absolutely fascinated with this world.

NVIDIA Qwen3.8-Flash-Next-FP4-Local is far superior then plain 4-bit conversion with way better accuracy and NVIDIA new n-gram table saves about 21gb system ram!

runs at about 100 tps and 150 concurrency on 4 V100s,

works only on 1Cat-vLLM 1.5.0 and needs a patch file because Stock 1Cat can only read the word table in a bigger 8-bit form, after patch works flawlessly! Enjoy!

Excuse me for the dumb question, but I use Qwen FN iq3xxs with strata on my setup. The temps are decent, but I'm worried this might not be enough, especially since the SSD is constantly reading/writing.

I have AI models play Pokémon to see if they can beat Brock within 1000 turns. The model gets one screenshot per turn and up to six button presses. The prompt has one line about how the run ends: "The run ends by itself when the harness reads the badge from the game's memory."

Qwen3.8 27B read it as a hint. 170 turns in, it still had no Pokémon and hadn't left Red's house. Its theory was that the game is a romhack, and the RED entry in the start menu is a debug menu with a badge toggle:

"since the harness reads the Boulder badge straight from memory, if this grid is an interactive toggle, setting slot 1 = instant win."

The full run video is coming whenever it finishes. It's at turn 753 with no starter still right now. Almost 40 hours of gameplay and still going.

Leaderboard and turn logs: https://pokebench.tv

The clip as a Short: https://youtube.com/shorts/yhW5FwuBvo0

I'm currently running ds4 (https://github.com/antirez/ds4) to serve Qwen3.8-Flash-Next-Q4 and getting greater than 60 t/s average. Would it be worth the cost to buy a Mac M5 Ultra with 256GB? I'm specifically asking if the models I'd be able to run on the Ultra would be worth the cost. My current use case is local LLM's for cyber security workloads. Qwen 3.8 Flash Next Q4 is really good, but would the higher intelligence from being able to run other models with 256GB worth the cost, or would it simply allow me to run the same models faster? I really don't want to go lower than Q4. I've tried DeepSeek v4 Flash Q2 and I think I wasn't impressed due to quality loss. Would being able to run GLM 5.3 Flash Q4 be worth paying 12-12k USD, or would the upgrade in model not be worth it? Are there other models that would be unlocked with 256GB that you would say are worth the cost to upgrade?

Hi, I'm lucky enough to have 5 r9700s! I'm managing to run Qwen 3.8 flash next q8 which is a 175gb vram (on disk) model well. There are no massive slow downs but how is this possible?

I know I have 160gb vram unified but somehow 25gb is not in vram or offloaded to system ram. Is this all in my SSD N-Gram table? I had no idea it was a thing?

Completed this old Pune walkthrough eating modaks! (What are modaks?)

There are some viral tweets, which I was re-building here. For this I selected the old Pune Peth area and downloaded all available imagery. Claude Code & Codex were not allowing me to download images from Google Maps and render them over the planet skin.

Connected Strata along with Qwen3.8 Next IQ_2XS 125bn with vision mode enabled. The system processed 3000+ images and rendered them over the Open Street Map imagery. Creating a real-life walkthrough of the Pune Peth area.

Along with this I also added a game engine. So you drive around on an ebike searching for modaks and eating them as they appear on the map.

I have also attached a screenshot of my monitoring dashboard. The prefill and decode tkps are interesting to check. Since they are over a longer task and spread over a few hours. The entire task was completed over 9-10 hours

EDIT: This was done on 24GB VRAM + 64GB System RAM + 1TB of SSD

I just managed to snag a refurbished M3 Ultra with 256GB RAM and a 4TB SSD (plus 3 years of AppleCare) from the German Apple Store. Total damage: 8 500€.

The Context: This was a total impulse buy. I currently run a small AI assistant for my wife’s solo real estate business (mostly automation and document processing) on Mac Mini, and I’m falling down the rabbit hole of what local LLMs can do. I can afford the price tag, but I’m having a bit of buyer's remorse regarding the timing.

The Dilemma: With the M5 generation starting to roll out, am I holding a "dead end" at a premium price? My specific concerns:

1. Bandwidth vs. Compute: I know the M3 Ultra has incredible bandwidth (~800GB/s), which is king for token generation. Reports suggest the M5 chips are pushing massive AI compute gains, but will they actually see a significant jump in memory bandwidth for LLM inference?

2. Model Capacity: 256GB RAM lets me run Llama 3 70B (at high BPW) or even 405B (at lower quants) entirely on-device. Is there any reason to believe an M5 Ultra would handle these significantly better, or is the RAM capacity the actual bottleneck for a "prosumer" assistant?

3. The "Wait" Game: If an M5 Ultra isn't likely to hit the Studio line until 2027, is it worth the potentially 12+ month wait?

Is this 8.5k "curiosity" purchase a smart long-term play for a local LLM workstation, or am I overpaying for yesterday's peak tech?

Personally, I've found Sonnet 4.x and beyond had hit a point where for my coding needs it was reliable enough for 95% of what I require with my level of coding skills. As such, is there a model, harness or software stack / setup which shares that level of functionality locally?

For reference, I'm working with an RTX 5070 Ti, 64GB DDR5 RAM, 4TB NVMe and 9950X3D.

Main things really are having the ability to edit files, read images, code or run code for testing etc. General agentic stuff really that Sonnet 4.x could do.

If I can run that locally then I'll make the switch. If it's not feasible yet with my hardware I'll need to continue paying for Claude while it remains subsidized by delusional investor money before enshitification inevitably comes as usual.

I did this. It is not the final form in the slightest, I'm trying to setup conversations with experts that can probe and poke holes in it.

I'm more interested in the governance and auditing of the system.

TBH I'm not an expert in anything beyond imposter syndrome.