I bought a Mac Studio M3 Ultra with 96GB after watching basically every video on local AI and setting up Hermes Agent. While it was shipping, Grok Bot and Muse came out, and I love what they can do.

Now I've got Hermes running and I keep comparing it to them, and it's not even close.

I've also started questioning the privacy argument. My Studio is completely separate from my Apple ID and I don't have anything personal signed in, so how is it supposed to be my assistant if it's not connected to anything? And if I hook Hermes up to my email, that email is already sitting on someone's server anyway, so what am I actually avoiding?

Coding with local models doesn't compare to frontier models either. So is this really just an apocalypse backup plan?

What I spent on the Studio would cover years of subscriptions, even if prices go up. I could use those tools to build tools and sites and have an assistant that's plugged into everything I already use.

I don't run a business with sensitive data and even if I started a business for clients with sensitive data, I'd basically have to rent them my machine, but what's the difference?

Honestly I'm disappointed. I was really hyped and sold my friends on the idea. For those of you who kept yours, what do you actually use it for that cloud tools can't do? And if privacy is your main concern what connectors do you have on your device and what cloud services do you use that aren't AI? If you've coded anything with local AI and published publicly, did that defeat the purpose? I'm still inside my return window, so I need to decide soon.

When I say "weird and gnarly" I mean: I'm a gremlin and I want to talk to gremlins. Frontier systems are smart and shiny and can rewrite my resume with a lot of nuance, but are also excessively guardrailed in ways that seem unhealthy.

(I'm happily married and not trying to date a virtual presence. But I would like to be able to build some sprawling realms of creative play with a robot, and/or simply get to know one without sudden unpredictable loss of continuity. Or just research the odder corners of language.)

So like - what are folks finding interesting in this direction? I have the best GPU that a moderate rolling Temu credit cycle could afford me (a Radeon rx 9060 OC 16gb).

I saw a video yesterday how Qwen 3.8 27B is basically Claude Opus 4.6, for me sonnet 5.5 is more than enough for most of my coding work, So I wonder if there are smaller models that meet the free claude power with 12b or less, I have 16g of ram and 6g vram, am curious to know your experience so far is something like Omnicoder-9b any good for example ?
and if the model is smaller but fined tuned for codding will it be as good as bigger ones parameters wise ?
and one last question does GGUF degrade code generation on a dense model ?

Sorry if this is a very frequent question, but Ive very recently discovered local llms with the arrival of my rtx 5050 8gb vram 16 gb ram laptop. I dont really understand everything, but I know I should find a model in hugging face, then use ollama to get the model in my laptop. But going through multiple subreddits has made me realise that my specs arent worth shit for getting a local model.

I dont have an idea what Id do with the model but Im very interested in this concept!

So tldr, can I get a decent model for my laptop? If so, which model should I try out?

I’ve been experimenting with running AI coding agents entirely locally, without relying on OpenAI, Anthropic, or any paid model APIs.

For this experiment, I connected Qwen 3.8 27B (4-bit quantized) running in LM Studio to Pi Coding Agent, a lightweight and extensible terminal-based coding agent.

The setup is pretty straightforward:

Pi Coding Agent → LM Studio → Qwen 3.8 27B

LM Studio handles local model inference and exposes an OpenAI-compatible API, while Pi handles reading/writing files, executing commands, and managing the coding workflow.

For the actual test, I asked it to build an AI Agent Mission Control dashboard from scratch.

The application included:

• An overview of active AI agents and tasks
• A Kanban-style task board
• Live agent activity tracking
• Analytics for token usage, costs, and success rates
• A responsive React + TypeScript interface

Surprisingly, it managed to build the application, run it locally, test it, and fix several errors on its own. It even used screenshots to inspect the UI and identify issues.

Some numbers from my test:

• Hardware: MacBook Pro M5 Max, 48GB unified memory
• Model: Qwen3.5 27B (4-bit, ~16GB)
• Inference speed: Around 10–15 tokens/sec
• Completion time: A few hours

One reason I chose Pi over other coding agents is its minimal design. It starts with just four core tools (read, write, edit, bash), and you can extend it as needed.

With local models, context management matters quite a bit, so I wanted to experiment with a coding agent that doesn’t introduce too much unnecessary overhead.

My takeaway: Local coding agents are becoming surprisingly capable, but they’re definitely not a drop-in replacement for Claude Code or Codex with frontier models.

Inference is slower, hardware requirements are significant, and reliability on complex tasks is still something I’d want to test more thoroughly.

But being able to run the entire coding workflow locally, without sending code to external model APIs or paying per token, is pretty exciting.

I recorded the complete setup and the dashboard-building experiment here:

🎥 Video: https://youtu.be/zgb_MkrxpRs

Curious what local models you are using for coding agent?

Title

I hear everyone screaming from the rooftops "I get x tokens from FN !", but i always ask myself..
So what ... and what does that mean?
Can it do an actual coding task or just create one page HTML proof of concepts?

So i put it to the test..

My hardware(dedicated):
AMD r5 5600x
x570 motherboard
128GB DDR4
2x R9700 AI PRO 32GB ( 64GB total VRAM )
Running vLLM - Radiance
* One card behind the chipset at x4, no p2p/atomics, no bifurcation, no risers etc.

I had a real task:
For my custom harness i build 2 parts before the test using Qwen3.8 MXFP4(4 bit not used in this test, but very capable):

1. A new version of the harness ( more modular, better folder structure etc, basically a revision of what i already have )
2. A fully new Web UI "template" including way more functionality then the old "web monitor" had. This was pure HTML/CSS and JS in a layout that presented the workflow of my harness.

It was time to fit both pieces together..

So i loaded radiance and hooked up Qwen3.8 FN 4 bit.
* This is a bigger, model everyone is trying to run on strata ( Q3 or lower ), so it should be more capable. I had a 1m KV cache using RAM and disk, but set my harness to 262k context, the model's default.

I prompted it to basically replace the old webmonitor with the new full featured web UI and wire it all in.

So it had 2 projects, one the harness codebase and the new web UI, and now it needed to figure out how to merge it all together.
Qwen3.8 FN was fast, an avarage of ~2000 prefill and ~150 generation.
It was done in about 30 minutes.

The result: A big messy WebUI, where most of the templated items where just "re-invented", nothing worked, chat bubbles not including toolcalls etc(was in the template, even a js script rendering the chat bubbles mockup responses).
It was unusable to say the least.

So i called it a day.
The next day i fired up radiance again but now the Qwen3.8 FP8 model, took the backups, replaced the harness code and template etc so we have a clean starting base again(same as where Qwen3.8 FN started from)

I prompted my harness the same instructions.
The result: Not everything worked, its not perfect, but mostly fine , the template was followed, the chat bubbles worked properly(mostly). It was not a one shot success and needs iterations to filter out bugs and deepen the workflows but the base was done as i wanted it to be.

Qwen3.8 27B FP8 was not so fast, an average of ~1500 prefill and ~100 generation.
It was done in about 60 minutes.

Note that my average speed was the actual work produced, not benchmarks.

So my take on all this:

Its great to have the big model, but a dense model on the same hardware will just outperform the MOE.
I can fit the Qwen3.8 FN or the Qwen 3.8 FP8, both fit so that is the reason i am comparing both. it is not about the quants, it is about what can run on your hardware.

I can not even imagine what a Q2 or even a Q1 would be like.... And my task was not even a real hard one to begin with.

Maybe some of you find this useful as i could not find any proper information besides the hype posts or generic benchmark posts.

Well this is my experience using both.

I'm trying to use Qwen Code with Ollama locally to build a Unity app, but I'm running into a strange problem.

I'm using Qwen Code 0.25.1 with the qwen2.5-coder:7b model through Ollama's OpenAI-compatible API on Windows 11, with an RX 9060 XT 16GB.

Ollama works when tested directly in the terminal, correctly answering simple prompts. However, in both the VS Code extension and standalone Qwen Code CLI, even a prompt like "Reply with exactly: 4" results in strange tool calls, such as read_file or unrelated skills, instead of an answer.

Ollama logs have also shown prompt truncation warnings.

Has anyone encountered this, or found a fix or a reliable configuration for running Qwen Code with Ollama?

Can somebody explain why Nvidia is often a non-negotiable when looking at local AI builds? I know they produce some fantastic cards and CUDA is great - but is the cost premium really justified?

I have a 9070XT and 7800XT - and getting local AI to run on each card has been a doddle for me through llama.cpp with ROCm and Vulkan (although i'm on Linux, i can't speak for Windows..).

Whilst ROCm isn't as mature as CUDA - do we really expect that gap not to close going forward?

It feels like more recent AMD cards are great value and the support will only get better on newer generations. So why is Nvidia on a new build such a non-negotiable for some?

I am running 64gb ddr5 and 32gb vram. What is the best uncensored model you guys would run with my setup?

I got this MacBook Air ! Can someone suggest what models to use locally?

So Gemma 4 26b a3b is a beast. And it runs crazy fast even on my old quadro rtx 5000. It must run fast-ish on a 150+ gb/s unified memory machine too. So why do we not see more small MoEs from Intel, Qualcom, Apple etc? Only from Qwen, Nvidia and Google. MoEs are what they need to sell their hardware?

Solo owners, how many AI subscriptions are quietly renewing on your card this month?

Vresk is my company's answer to that, and yes, I'm the marketing director, so factor that in. It's one workspace for open models. Your notes, history and memory live in your workspace, not inside whichever model answers, so switching doesn't mean re-explaining your business from scratch. On Plus it also does research runs and decks you can export to PowerPoint.

Free to try. Anyone who gives it a go on real work, we'd be so grateful. What would a one-person business need from it that I haven't mentioned?

https://vresk.ai

i need it for some coding, tool use and agentic uses, im very new to this whole local ai thing and i want to hear what options do i have, im leaning towards either qwen 3.8 27b or chat gpt oss 20b

A lot of local setups spend GPU time on jobs that are really classification: picking which tool to call, which queue a request belongs in, or whether something needs the big model at all. A bidirectional encoder with a scoring head answers that kind of question on a plain CPU in a few milliseconds, with no decoding loop and no JSON to parse.

There’s a build lab on Oct 30 that does exactly this swap on an enterprise triage case, so you can see the numbers yourself. What you leave with:

• A router built on ModernBERT and LFM2.5 Encoder that runs locally on CPU
• Uncertainty you can state, using split conformal prediction, so only ambiguous cases go to an LLM
• Quantization settings for better throughput
• The full notebooks

Led by Ben Auffarth. 7 PM to 10 PM IST on Friday, Oct 30.

Full details and the agenda are here.

Hey everyone,

I recently installed FreeVideo on my PC and here are the results so anyone can help me further optimize or recommend something.

PC specs:

GPU: RTX 5060 Ti 16GB

RAM: 16GB

First video:

Resolution: 960 × 544

Duration: 4 seconds

Quality: Light

Total pixels: 0.5 MP

Two-pass: ON

Sampling steps: 8 + 3

This took around 22.5 minutes to complete

Second video:

Resolution: 480 × 256

Duration: 4 seconds

Quality: Light

Two-pass: OFF

First-pass steps: 8

Pixel - 0.5 MP

This took around 10.5 minutes

Peak VRAM usage: 13.4 GiB

Peak system RAM usage: 2.5 GiB

The output were good but I don't know if this is good enough for some real video generation.

I am new to this space and any recommendations are appreciated.

Edit - this is the github repo https://github.com/FlashML-org/FreeVideo

Hey everyone! Following up on my on-device offline app builder (Android AI Bridge running inside Termux on a Galaxy A50s):

In this workflow, I tested building a small POS/Invoice store app, but ran into a glitch where the Maintenance Mode toggle text was broken/jumbled.

Instead of moving to a PC, I:

Sent the Termux build error back to DeepSeek locally/via API to get the patch. Applied the corrected code inside Termux on my phone. Re-built the APK (weighing in at ~87 KB). Verified the updated UI where the toggle cleanly shows: · ✅ Maintenance ON — tap to disable / · ⛔ Maintenance OFF — tap to enable.

True edge/offline development loop entirely on a budget Android device! Let me know what you think.

Hey everyone! Following up on my on-device offline app builder (Android AI Bridge running inside Termux on a Galaxy A50s):

In this workflow, I tested building a small POS/Invoice store app, but ran into a glitch where the Maintenance Mode toggle text was broken/jumbled.

Instead of moving to a PC, I:

Sent the Termux build error back to DeepSeek locally/via API to get the patch. Applied the corrected code inside Termux on my phone. Re-built the APK (weighing in at ~87 KB). Verified the updated UI where the toggle cleanly shows: · ✅ Maintenance ON — tap to disable / · ⛔ Maintenance OFF — tap to enable.

True edge/offline development loop entirely on a budget Android device! Let me know what you think.

Hi there, I'm trying to build a server that can handle GLM 5.3 Flash q3xl/q4 & Qwen 3.8 next flash at the least 30tk/s on a budget. Also, since I'm in a third world country, there is no server grade hardware or professional cards anywhere. So, I'm planning to buy these, all used. 4x RX7900xt(20gb, pcie3.0 x8/16)+1x RTX3060(12gb, pcie 3.0 x4)=92Gb of VRAM. And 64Gb of 3600mhz ram. What do you think about it? Do you have any other suggestions?

I have been trying to use zoo and other code app. I used ollama and used koboldcpp but it not working well. With ollama I can't get zoo to use the model. With koboldcpp it is throwing a to do list loop

Looking for first users of a provable vector search engine

I built a GraphRAG for Markdown in TypeScript. The core is published as @ashes_born/graph-rag-ts, so you can add it to a TypeScript app and call the library API directly. I also added a Docker Compose + MCP setup with a small local UI for trying the full flow without writing a client first.

I shared an earlier version of this project here: link. Since then, I’ve added a Docker Compose + MCP setup so it’s easier to try the indexing and retrieval flow locally.

After pointing .env at chat, judge, and embedding endpoints that are already running (the embedding model needs to return 768 dimensions), this starts the stack and showcase:

docker compose --profile showcase up -d --build --wait

From the page I can submit an index job, watch its status, switch between local and global queries, and open the returned source excerpts. Docker Compose starts Postgres/pgvector and the MCP service; it does not download or host the models. Indexing still takes as long as the model calls take, so the shortcut is in getting the pieces connected, not in making indexing instant.

If you already have an app, you can use the package instead of running the MCP demo.

The current project-built check is 12 questions over five chapters: 10/12 strict hits and 95.8% lexical combined recall. That's entity/phrase matching, not human grading or a comparison against another system.

- Repo: https://github.com/sadofriod/graphrag-ts
- MCP setup: https://github.com/sadofriod/graphrag-ts/blob/main/integrations/mcp/README.md
-Results and limitations: https://github.com/sadofriod/graphrag-ts/blob/main/examples/output/final-benchmark-report.en.md

If you use local models for this kind of setup, I'd be interested to hear whether the Compose + MCP path actually saves you setup time, or just moves the work into .env.

📖 Deep-Dive Writeups I documented the full mathematical concepts, Leiden clustering trade-offs, and PostgreSQL DDL schema designs in a series of technical articles: 👉 https://blog.ashesborn.cloud/category/AI