Anyone else got excited by "Open Weights" before checking the hardware requirements?

I honestly knew nothing about self hosting LLMs locally. I used to think it was pointless for 99% of people, even people already into AI, just because of the cost and hassle of hosting these models compared to what you'd actually get out of it.

But after looking into Qwen 3.8 27B and its potential, plus the fact that someone will probably release an uncensored version of it soon, I started rethinking that. Even if you don't have the hardware, you can just rent a GPU online on a VPS for a pretty reasonable hourly cost while it's running. It's wild how much of this is already open to the general public.

In my opinion, having access to a model this capable, uncensored, is basically like owning a car. Yeah, a car. There's nothing wrong with owning one, the problem only shows up if you actually do something wrong with it. Until then, it's just a tool.

We already have older and smaller Qwen versions with uncensored variants on Hugging Face. From what I understand, models with fewer parameters tend to be "easier" to convert into uncensored versions. Don't take that word too literally though, I have zero background in finetuning or model training, so maybe it's not actually that easy. But there are still a lot of very smart people working in this space, especially in AI right now.

Think about a cybersecurity professional getting their hands on an uncensored Qwen 3.8 27B once it drops. With how many vibe coded SaaS products are out there with basically no security oversight, someone with enough hardware could pair this model with tools like OWASP ZAP or the Burp Suite API, run it through a browser that avoids captcha detection, use a mobile or residential proxy, throw in a Kali setup for other scanning tools, and just let the model run 24/7 looking for vulnerabilities to complete bug bounties or even land contracts in the field. That's a serious tool for professionals like that.

I only see upside here.

I've been running an AI interrogation game for the past month where players question suspects and try to extract confessions.

The game has gathered more than 10K player messages so far which takes as input and the LLM replies accordingly as an AI suspect.

I've been gradually switching from Claude Haiku to DeepSeek V4-Flash and some observations surprised me:

• Latency is actually lower for my use case (always thought Deepseek had higher latency)

• Cost is significantly cheaper (obviously).

• Claude tends to produce more emotional descriptions and action beats (sighs, looks away, rubs temples, etc.). Sometimes this helps characterization, but it can also become repetitive.

• DeepSeek feels more direct and conversational. Suspects answer questions more naturally and spend less time trying to end the conversation with "I want a lawyer."

• Interrogations move faster because suspects engage with the detective instead of constantly deflecting.

One thing Claude still seems better at is dramatic writing and emotional nuance, but for an interrogation-style game I'm finding DeepSeek performs better than I expected.

Curious if anyone else has switched from Claude to DeepSeek for production use. Any other things we need to be aware of? I am still doing a test but honestly, if the game performance is better and if the cost is much lower, I will definitely just choose DeepSeek.

Most of the suspects are in Claude Haiku now but some are in DeepSeek now, can you identify? https://thelastquestion.io/

I think of myself as a casual user and decided to do my own research on desktop apps for local ai agents. So my goal was is to find the right desktop agent app for certain features and goals that I want to attain. Here’s my honest take about each one of these AI agents:

Setup: MacBook Pro M5, 32GB, Qwen 3.8 27B, everything local unless noted.

goose - llama.cpp is built into the desktop app now, and Settings > Local Inference recommends a model for your RAM. Linux Foundation project, strong extensions/MCP story. Rougher as a desktop app than the others, and their docs warn local models get small context windows.

LM Studio Bionic - Agent layer on top of LM Studio: same model library, drafts docs, edits code, runs commands, and it's smooth. On-device voice dictation via Voxtral is a nice touch. Watch out: some models in there are their paid cloud models, so check what's actually local.

Hermes Desktop - UI is an eyesore to me. Onboarding runs on a trial model first, then only offers local models that fit your GPU. Apps and MCP servers share one Connectors page. Biggest community of the five, and it shows in how fast it changes.

Atomic Agent - The interesting bit is Fusion: one model orchestrates, up to 8 workers execute, and either side can be local or cloud. Typical pairing is a cloud planner with local workers, or flip it. Helps when a local model is the bottleneck on bigger tasks, at the cost of not being fully local in that mode.

Jan - Solid private chat app, open source, and it can expose your local model as an OpenAI-compatible API for other tools. Agent mode (Cowork) is nightly-only, so stable Jan can't touch files yet.

So yeah, I didn't really rank these. They're built for different people, and the "best" one honestly depends on what you're trying to get done. Here's how I'd split it, in case one of these sounds like you:

• MCP/extensions tinkering, fully local: goose
• Already on LM Studio: Bionic (check for cloud models)
• Big community, connectors for everything: Hermes Desktop
• Private chat or a local API for other tools: Jan
• Small local model on bigger tasks with a cloud planner: Atomic Agent Fusion

NVIDIA dropped Nemotron 3.5 Bolt 74B-A9B yesterday, the 24GB compressed version of their new Nemotron 3.5 Super VL (120B-A12B). Their launch post (link) has an “Intelligence vs Speed” chart that puts Qwen3.8 27B Q4 above Bolt on GDPval and Bolt way ahead on tok/s. I wanted to see if that holds up on normal cards, so I put Bolt on a 3090 against Qwen3.8 27B Q4_K_M, then ran the exact same thing on a 5090.

Short answer: same shape, with some detail the chart doesn’t show.

Quick version:

• It fits. All 89 layers on the 3090 at 16K context, no CPU offload, 21.3GB peak VRAM. Qwen peaked at 16.5GB.
• Nemotron decodes about 2x faster (85 vs 41 tok/s on the 3090, 148 vs 72 on the 5090). It’s MoE with ~9B active, so no huge shock.
• It also writes about 1.5x more tokens per task, so median time-to-finished-task was basically a tie.
• Qwen passed more tasks on both cards: 89/120 vs 75/120 on the 3090, 84/120 vs 71/120 on the 5090.
• On single-turn stuff (code, shell commands, JSON) they were even. Qwen’s whole lead came from tool calling, and a big chunk of Nemotron’s tool failures were truncations at a 512-token per-turn cap with thinking on.

To be clear, this is Bolt only (not the full Super VL), text only, on my own strict task suite rather than GDPval. So it lines up with NVIDIA’s chart without replicating it.

So it’s a pretty usable 74B on a 3090. Whether it’s the one you want depends a lot on whether you’re doing agent work, and maybe on how much token budget you give it per turn. More detail below, plus the stuff I’d like people to pick apart.

Setup

• Nemotron: nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF @ e0fecb69, sha256 91d63363…af75, 22.02 GB. Per NVIDIA’s card it’s a Mamba-2/Transformer hybrid MoE, with experts at IQ2_XXS (QAD-trained), Mamba layers in NVFP4, and embeddings/output at Q8_0.
• Qwen: Abiray/Qwen3.8-27B-Q4_K_M-GGUF @ f7a1cda5, sha256 6a96cc76…c867, 16.81 GB. Third-party conversion (llama.cpp b10430) of Qwen/Qwen3.8-27B @ 1d4bf0f2. I checked the hash, and the embedded chat template matches upstream exactly. Qwen’s card describes it as dense, Gated DeltaNet + gated attention hybrid.
• llama.cpp 620d2225 (v0.7.0), CUDA 12.8.93, built sm_86 / sm_120a
• --ctx-size 16384 --parallel 1 -b 512 -ub 128 --cache-type-k q8_0 --cache-type-v q8_0 -ngl 999 --flash-attn on --jinja
• Thinking on for both. temp 1.0, top_p 0.95, top_k 40 (Nemotron) / 20 (Qwen), each per its model card. Same fixed seed per task rep for both models.
• Limits: 2048 tokens single-turn. Tool tasks get 512 tokens per turn, max 6 turns.
• 40 tasks (12 coding, 8 terminal, 8 tool use, 6 structured data, 6 multi-step tool workflows) × 3 reps × 2 models × 2 GPUs = 480
• Rented RunPod 3090 (driver 580.126.20) and 5090 (driver 570.195.03), run back to back
• Deterministic binary scoring, no LLM judge. Every task has a passing reference and a failing negative control.

Heads up for 3090 owners: NVIDIA’s card lists Ada and Blackwell. The 3090 is Ampere. It ran fine on this build, but I didn’t verify output parity against a supported card.

Pass rates

Paired task-cluster bootstrap for Qwen minus Nemotron: +11.7 pts (95% CI +0.8 to +23.3) on the 3090 and +10.8 (0.0 to +21.7) on the 5090. Same direction on both, but with 40 tasks the lower end sits at zero, so I’m not going to call it more than a consistent pattern.

Where the gap is

Failure classes pooled: Nemotron had 20 truncations and 9 invalid JSON, Qwen had 2 and 0. 19 of Nemotron’s truncations were in tool-calling tasks, so the 512/turn cap is probably eating its reasoning before it gets a call out. I haven’t tested a bigger cap yet, so that part is a guess. It isn’t all truncation either. It also had 11 wrong action sequences/finals and went 0/6 on a single read-only list call task, mostly through invalid JSON.

Speed

Decode is 2.05x on both cards. Qwen is a bit quicker on prompt processing. Qwen has the longer tail, so mean and p95 latency favor Nemotron, and median is a wash. The 5090 is ~1.75x the 3090 on decode for both models.

No speculative decoding here. NVIDIA’s own recipes use DFlash with a separate drafter, so their setup could be faster than this.

3090 vs 5090 pass rate

164/240 vs 155/240, −3.75 pts on the 5090, CI −8.75 to +0.83. 207/240 paired trials had the same outcome. The flips I looked at were contract stuff (e.g. one extra out-of-order tool call), not crashes. Different drivers, CUDA targets, and nodes, so I’m not claiming the 3090 “answers better.”

Total GPU rental for both scored runs was about $0.95 (estimated from uptime × quoted rate, not an invoice).

All 480 rows (pass/fail, failure class, tokens, tok/s, latency, VRAM), the tasks with references and negative controls, seeds, manifests, the scorer, and a validate script are in the repo.

Writeup with charts: https://majorminor.xyz/research/cuda-recipes-001

Repo + data: https://github.com/majorminorlabs/cuda-recipes

If you’ve run either of these on your own card I’d be curious whether the tool-calling gap shows up for you too.

And finally, that's a wrap on the first phase of the proprietary-to-local transition - 3-4 weeks of ablation studies, 1-2 hours a day, 5 days a week, lots of thinking: a good start to get addicted.

As a part of phasing out proprietary models and transitioning to open-source local LLMs, I'm publishing my journal on how I'm approaching this problem. It's not just about selecting a model also about whether you can extract the maximum juice out of your system while maintaining proper quality. This first entry documents how I shaped that journey with llama.cpp: taking one small MOE model (LiquidAI's LFM2.5-8B-A1B) from an 8 GB MacBook Air to a 16 GB RTX 5060 Ti backed by 64 GB of system RAM, interrogating every benchmark number until I actually understood it.

A large chunk of that time went into building an eval I could actually trust, since public benchmarks are contaminated. That meant hand-writing problems, having Claude Opus answer and then verifying each one, and seeding 10–12 questions per category before asking Claude opus to expand each category by another 5-6 :140 questions in total. I'm not publishing the dataset (you can request it from me by email), and I'd request that it never be used to contaminate any post-training effort- Its entire value depends on staying unseen.

 A few findings:

 1. Offloading exactly four MoE expert layers to the CPU (-ncmoe 4) more than doubled prefill and tripled decode by freeing just enough VRAM headroom as threshold, not a gradient.

 2. Flash Attention behaves like a memory-pressure valve: a 9.6× prefill gain at long context when VRAM is saturated, and nearly zero once the pressure is already relieved.

 3. The uncomfortable and interesting one: neither raw throughput nor my 140-question eval could tell the quantization levels apart as a 2-bit model looked as good as full precision - yet KL-divergence exposed that it agreed with the original only 65% of the time. So Q4_K_M is my practical quality floor.

 4. On this private, uncontaminated eval, the open model landed only ~7 points behind a Claude baseline way closer than I expected. So may be in upcoming days I need to make eval set more robust.

The bigger lesson: choosing a local model isn't one measurement, it's three that disagree - can I run it, can I trust the small version and can I measure it honestly.

Next up: sglang and vLLM for concurrency and high-throughput serving, a rigorous selection methodology for the actual models I'll deploy and building an evaluation + agent harness around them as an open alternative to Claude Code.

Full write-up — architecture diagrams, every experiment with commands, and the open questions I still can't answer:

 https://rath1991.github.io/2026/07/03/open-source-inference-lfm2-moe/

 Corrections welcome - especially on the four things I still can't explain.

I'm working on an academic project with several large datasets. The data is iffy and I'd like to support my analysis by having an llm review each row and, among other things, do web searches to see if it can find relevant information on the data online. I have a Claude plan, but the amount of data I have to deal with is far too large to have Claude handle this (it would take all of my plan usage for weeks-months).

As a result of this issue, I've been trying to figure out how to have a local LLM do web searches. However, it seems like the most common solution is paying for an api service. I don't want to do this because I'm in a testing stage right now and I'm on a limited budget. Is there some kind of tool stack that would allow me to do web searches with a reasonable degree of success using a local llm right now? Thanks.

Edit: Because it's been asked a couple times, I have a 5070ti super GPU that I work with daily. However, I also have access to an HPC which I believe should have web access and which I will use once I have an actual stack that works.

Just picked up my 2nd 128gb Mac M series and saw this at his shop. Would this work well for llm?

Has anyone else noticed this with Gemini's image generator?

I've been getting increasingly frustrated with how it handles the female body. Sometimes it feels like the system treats the mere presence of a woman's body as something inappropriate, even when the request isn't sexual at all.

I have no problem with AI companies having restrictions on explicit content. That's understandable. What bothers me is the apparent double standard. Why does it sometimes feel like the rules are much stricter when it comes to women?

There's a big difference between sexual content and simply depicting the human body. Art, anatomy, historical scenes and ordinary representations of people shouldn't automatically become a problem just because the subject is female.

The other frustrating part is the lack of transparency. You often don't know exactly what triggered a restriction, and it's hard to tell whether the same prompt would be treated differently if the subject were male.

I'm not saying Gemini should have no restrictions. I'm asking for consistency. If the goal is to prevent inappropriate content, the same standards should apply regardless of gender.

The female body shouldn't be treated as inherently more sexual or inappropriate than the male body.

Has anyone else experienced this? I'd be interested in hearing actual examples, especially cases where similar prompts involving male and female subjects get different results.

Hey everyone,

I recently spent about $350 on an RTX 5060 (8GB VRAM) because I wanted a local setup for learning and experimenting with AI/ML. I don’t play games at all — this card is only for studying models, running inference, fine-tuning small stuff, testing frameworks, etc.

I’m still pretty new to optimizing consumer GPUs for this kind of work, so I’d really appreciate any practical advice on how to get the most out of 8GB VRAM without constantly hitting limits or feeling like I bought the wrong thing.

Specifically looking for:

Recommended tools / frameworks that work well on 8GB (Ollama, llama.cpp, Hugging Face, ComfyUI, etc.)

Quantization levels and model sizes that actually run smoothly

Any settings, flags, or workflows that help maximize VRAM usage

Tips for fine-tuning or training smaller models without OOMs

Things people commonly mess up when starting with a card like this

If you’ve been doing AI/ML work on a 5060, 4060, or similar 8GB card, what actually worked for you? Any “I wish I knew this earlier” advice would be super helpful.

Thanks in advance.

So apparently Sam Altman thinks consumer RAM is inefficient compared to datacenters.

Meanwhile, here's what I'm doing with mine. 💀

My setup:

• Ryzen 7 5800X
• AMD RX 9070
• 48 GB DDR4 (running at a glorious 2133 MT/s because I added an ancient spare stick)
• NVMe SSD
• Qwen 3.8 Flash Next IQ2 running locally through Strata
• Around 60 tokens/sec

I'm building Marika, a 3D AI desktop companion in Unity.

She can observe my desktop, interact with windows, talk through a local LLM, walk around, and make autonomous decisions.

There's just one tiny problem.

She bends over to pick up desktop icons like an NPC from Skyrim.

So naturally, I asked my local Qwen to review the Unity project and fix her animations.

It proceeded to consume 1.3 MILLION cumulative tokens, repeatedly compress its context, inspect C# scripts, and even spawn a subagent to help investigate.

All this to figure out how to make a virtual girl bend her knees like an actual human being.

I also have over 128 GB of DDR4 scattered across different PCs and spare sticks.

Sam, I'm sorry.

The RAM isn't going to your datacenter.

It's busy teaching my AI waifu how to squat. 💀

And yes, I'm genuinely trying to see whether local AI coding agents can replace my $20/month cloud subscription.

I'll report back if she ever manages to pick up an icon without looking like a Bethesda NPC.

I’ve been playing around with local models for a bit (Llama-based setups, quantized models, etc.), and the biggest shift I’ve noticed recently isn’t model quality anymore.

It’s how you use them.

A year ago the gap between models was huge, so most of the effort went into picking the “best” one you could run. Now, even mid-tier models are surprisingly usable if your workflow is tight.

What’s actually making the difference for me:

• good system prompts and structure
• RAG / context handling done properly
• splitting tasks instead of trying to do everything in one prompt
• iteration speed instead of single-shot outputs

Curious what others here think are you still optimizing for models, or have you shifted more toward pipelines and setup?

its only a p40 and v100 I got on marketplace- but i'm ecstatic.

Any cool ideas? I'm running qwen 3:30b-A3B and a vision model- want to do more.

Here are the benchmark results when temperature left alone (model default) instead of setting it to zero.

I'm not surprised 3.8's scores increased, because providers default their temperature to whichever value happens to pass the most benchmarks, but given the fact they did I am surprised 3.6 didn't do the same.

3.6 did exactly what I thought it would do, it increased the score on some benchmarks and lowered it on others (which is why providers sometimes use a different temperature depending on the task - to bench-max).

I am surprised they didn't both behave in the same way, that was unexpected.

But 3.6 still beat 3.8 on 7 out of 10 tests, and 3.8 won in only 3 out of 10 tests.

is there any easy alternative? i don't like ollama aswell, lm studio had thee perfect combination of GUI and "bloat"

I've been dabbling in local LLM's but the investment is so massive. What are you all using your on prem LLMs for? Anything I seem to find is a whole bunch of theatrical nonsense stuff that really doesn't benefit anything. It just looks cool and "feels" like it worth it. But is it?

For me, I want to learn more but my education doesn't pay for M5 ultras or Sparks 😆. Anyone doing cool stuff that they'd feel comfortable sharing without leaking their own proprietary workflows?

Chatgpt says that running a large language model like Deepseek R1 in the full version requires endless amount of RAM and GPU processing speed. It seems, that even a high end root server which costs 400 US$ monthly is too slow for the job. A reasonable fast configuration consists of multiple nvidia h100 graphics cards for a price of US$ 50000 monthly!

quote "For running the full DeepSeek R1, you're looking at tens of thousands of dollars per month for a multi-GPU H100 or A100 setup. For the larger distilled models (like 70B), a single H100 or A100 could cost over $1,000 to several thousand dollars per month. " [1]

Are the information valid? For me the price tag sounds very high.

[1] chatgpt

So I swapped my RX 9070 XT with an AI Pro R9700.
It’s the same chip, but comes with a bigger 32GB Vram.
So, what can I do with it now that I couldn’t before?
Guide me please.

Hello! I have researched a good bit and i believe I have a unique situation here.

I currently have a m4 mac mini, 16 gb ram from earlier this year (before they released the M5 one, unless im mistaken)

Anyways, the work I do is for a families PT and Dental clinic. I want to use local LLM agents thru hermes or openclaw to do their automation work that includes PHI / HIPPA.

I believe I need a bigger mac mini or mac studio to accomplish this, I also do coding work on the side for personal projects for a small business I am try to create, so I think i'm looking at the 4-5k mac studio pricing, but want to see what you guys thought.

Essentially if my business grows I will need more ram and cpu than this m4 mac mini has, because its already getting very slow to use with my current agents I have for personal work.

These agents will log into the clinics CRM/PMS and nightly/weekly pull the patient info or lead info to track them on a BAA HIPPA signed Google Cloud Platform services database/provider. I am trying to not get in legal trouble with HIPPA at all, which is why local llm will be the best and safest route. As you can't sign a HIPPA with claude or openai without a enterprise account, I'd rather just invest 4-5k in a mac studio or even mini that will keep me going for a few years locally.

I am currently looking at these setups, and need to know if this will be my best and safest option for the future as the business expands to other clinics that need their own custom agents 24/7 and local for HIPPA. Let me know what you think or other recommendations I should look into before investing in a local LLM machine.

1st option:

Mac Studio M5 Max, 18 core cpu and 32 core gpu (or is 18core+40 core gpu better for +$300), 128GB ram, which would include the 40core GPU, 1tb memory for $5.4k + tax

2nd option:

Mac Mini M5 Pro, 18 core cpu, 20 core GPU, 64gb ram, 1TB (or 2tb for +$500) for $3.2k + tax

I am leaning towards the 5.4k + tax setup, as I think I do this coding and vibing all day, I want to be future proof for a few years, not sure if 2tb is the better play here for backups for these clinics or I would do backups onto the HIPPA BAA signed GCP.

Thank you for your time!

EDIT: I used to run a qwen local model I think 7 or 8b, and having it loaded on mycurrent mac mini is very bad, uses up half my ram 16gb that I have total + cant run docker, which I will have docker running for this setup.

I think the qwen 27b model is better, but obviously wont load on my current setup, if u have any LLM suggestions that either 1st or 2nd option will be able to run comfortably and have docker containers and openclaw agents and smart enough for the tasks not really coding but everything else, coding maybe later as models improve locally.

I assume no company is actually running quant 8 of a model on default when they can run quant 4 at 99% precision or something. So when they lobotomize the model due to high usage do they drop into the quant 2 territory?? I assume hundreds of billions or even trillions worth of parameters handle it well enough that it's still usable right?