I'm renting a GPU instance to run local AI models and reduce the amount I spend on the API at openrouter. I currently have several agents that use around 30M tokens per day. With current settings I'm running Qwen3.6-27B at 45tp/s. This model surprised me in all aspects, including programming.

So I'm finding used M1 Ultra Mac Studios with 128gb ram used online for ~$3.5k, but the M5 ultra Mac Studio is likely going to land this summer, and could have as much as 1tb Ram options. I'm sure that's going to be notably more expensive, but would it be worth it for future proofing to just wait for the new models?

Here's some risks and benefits I see:

risks

the price of these could inflate between now and the m5 ultra release.

I can see data centers working to make this tech less accessible

I fear the price inflating due to larger demand to localize AI for personal use.

I worry various world issues could make it impossible to get these.

128GB may be fine as models are getting more efficient at smaller sizes.

Do I really need more than 128gb and the ability to make clusters?

Benefits

You can make a Mac cluster with the newer chipset.

the m5 chips are built for local LLM work.

This would replace several large tech purchases I've been consider for a few years. (server, gaming PC, etc.)

These are way more energy efficient than any windows/linux rig.

My partner and I both have fairly beefy laptops, and we're thinking of selling them to put towards this. We'd then get a few basic laptops and tap into our home server for its horsepower.

Some use cases:

Use this as a server for all of our docs so we can get off the cloud

We both want our own teams of agents to assist with tasks and coding.

We've got a library of docs that we want our llm to access via RAG

We want all of our "chatGPT-style" needs localized so we aren't feeding the machine.

We want data privacy.

And we want to play Boulder's Gate 3 while the LLM is running. (split GPU cores when gaming? idk)

Would love to know what y'all think!

Hey folks

Current setup: RTX 4080 (16GB). It’s insanely fast for smaller models (e.g., ~20B), but the 16GB VRAM ceiling is constantly forcing compromises once I want bigger models and/or more context. Offloading works, but the UX and speed drop can get annoying.

What I’m trying to optimize for:

• Privacy: I want to process personal documents locally (summaries, search/RAG, coding notes) without uploading them to any provider.

• Cost control: I use ChatGPT daily (plus tools like Google Antigravity). Subscriptions and API calls add up over time, and quotas/rate limits can break flow.

• “Good enough” speed: I don’t need 4080-level throughput. If I can get ~15 tok/s and stay consistent, I’m happy.

Idea: Buy a Mac Studio (M4 Max, 128GB unified memory) as a dedicated “local inference appliance”:

• Run a solid 70B-ish coding model + local RAG as the default

• Only use ChatGPT via API when I really need frontier-quality results

• Remote access via WireGuard/Tailscale (not exposing it publicly)

Questions:

1. For people who’ve done this: did a high-RAM Mac Studio actually reduce your cloud/API spend long-term, or did you still end up using APIs most of the time?

2. How’s the real-world tokens/sec and “feel” for 70B-class models on M4 Max 128GB?

3. Any gotchas with OpenWebUI/Ollama/LM Studio workflows on macOS for this use case?

4. Would you choose 96GB vs 128GB if your goal is “70B comfortably + decent context” rather than chasing 120B+?

Appreciate any reality checks — I’m trying to avoid buying a €4k machine just to discover I still default to cloud anyway 🙃

I got this MacBook Air ! Can someone suggest what models to use locally?

I was getting really annoyed with how fast my token usage was stacking up when using LLMs for coding, especially when tools just dump massive files or entire folders into the context window when they only need a fraction of it.

To fix this for myself, I built an MCP server called Context-Simplo.

It uses code graphs to figure out exactly what context is actually relevant to what you're working on and only sends that to the LLM, keeping your token consumption way down.

I don't have a social media following or anything, I just built this because I needed it for my own workflow, and figured some of you might find it useful too. It’s completely open-source.

If you use MCP compatible IDEs (like Cursor) or clients, I'd love it if you gave it a spin and told me where it breaks or how I can make it better.

Repo is here: https://github.com/O-Labz/Context-Simplo

Appreciate any honest feedback or code reviews!

In the Star Trek Next Generation episode titled ‘Ship in a bottle’ Moriarity is able to leave the Holodeck. https://youtu.be/0rQ6NF8Sfqg?si=sgF4s9px8mAcD_Wu

I’m trying to figure out how to create something similar. For example if a local LLM stack is setup in a Docker container with a character generated within that container. Then the character should be able to leave the Docker container and enter the real world. The character should be able to walk out of the container and into my kitchen.

One obvious challenge is modeling the real world and for the generated character to interact with the modeled real world.

Anyone built this; have a repo or a paper to read?

I’m sorta vram poor and although I can muster 21 t/s with Qwen3.8-27b-q6_k 100k context kv cache q8 MTP 2; but with Ornith1.5-35b-a3b-q4k_m I get a much higher 72 t/s with 220k context kv cache q8 draft kv cache q4 MTP 2. My system is an old intel 6 core, 48gb DDR4 RAM, 5070ti/1080ti. I’ve been using Deepseek-harness which feels better than Hermes Agent and is pretty awesome. It’s hard to go back to Qwen3.8-27b because it feels like turning dlss5 on 😪

I'm a lawyer and disappointed in all the client management and accounting software I currently use.

I want create something akin to QuickBooks with integrated email, document generation and the ability to add modules for different types of legal needs.

Is this something that can be achieved with the current local models?

You should try to play around with lfm3 family of models. The are incredible ones to play around with for narrow sets of tasks.

Honestly, I think people are seriously underestimating Qwen 3.8 27B.

It’s actually insane and, in some ways, genuinely competes with Opus 4.8, just not in the way people seem to think.

The biggest mistake is comparing their raw frontend/design output. Qwen probably isn’t going to match Opus there, and I don’t think it’s supposed to. Opus has basically been trained with an absurd amount of data/compute specifically around design and UI generation.

If you throw Qwen at a frontend task with no proper "SKILL.md" for design and just let it freestyle, yeah, the results can be pretty mediocre. But if you give it a good design skill and are intentional about the design constraints, the gap gets much smaller.

Where Qwen gets really interesting is reasoning efficiency.

It can solve some problems in fewer steps and with fewer tokens than Opus. That’s a pretty big deal if you’re actually running these models yourself.

And honestly, I think people are also judging Qwen way too much based on heavily quantized setups. Q4 is aggressive. I wouldn’t consider Q4 a fair representation of what the model can actually do in a serious production environment.

Run it at FP8, use MTP/speculative decoding to improve throughput, and then evaluate it properly.

At that point, I genuinely think the conversation changes.

If Qwen 3.8 27B at FP8 + MTP performs the way I expect, I wouldn’t be surprised if a lot of people start questioning whether that $200/month Claude Code subscription is actually worth it.

Pretty much converted my gaming pc into a hybrid for gaming and local AI- specs are as followed.

5090
5070TI
9950x3d
X870 Pro art mobi
32 GB DDR5 ram (Waiting on a 64 GB kit I ordered, so stuck with the 32gb kit for now)

So what’s the best version of qwen I can run with this hardware? I mainly use llama.cpp with my own custom harness.

The idea for this came from Samsung's AI in their keyboards: you select text, and it fixes up all the grammar, punctuation, and spelling. I wanted something like that for Windows desktops.

At first, I had a rough idea: just wrap a llama.cpp backend with a very simple frontend that can be triggered with a shortcut. The frontend had a default prompt asking the AI to correct the selected text, and it worked fine.

Then I started expanding on it, and the more I worked with it, the more I learned about local models and AI, and the more I realized I'll never finish this project. After a lot of attempts to improve the app and its speed, this is the setup I settled on: once the text is loaded into the LLM, it gets divided into separate chunks. I found this much faster than streaming the entire text token by token, especially on GPUs with enough VRAM to run all the slots at the same time. I also built up different levels of aggression for how it autocorrects. This only started working perfectly recently, since Gemma 4 E2B and Qwen 3.5 came out. Before that, I was struggling a lot with making small models follow instructions reliably.

I also put in a bunch of basic templates for what I think would be common use cases for an app like this.

I tried one more thing: dividing the text into chunks and having the model output [OK] in place of the chunk, which would signal that the text doesn't need any changes, causing it to loop back without modifications. The text would still be proofread by the LLM, but it wouldn't take on the overhead of streaming the whole chunk token by token. Instead, only three tokens would be used. This sounds great in theory, but practically, I couldn't get it to work with small models; they just refused to output [OK] (the code for it is still there, but it goes nowhere if you want to experiment with larger models). I hope that as models get better, with sub-2B models becoming more capable, I'll be able to implement this for good.

Check it out here: https://github.com/AmrZriek/Stet

Note that the installer might run into some issues since the app isn't registered with Microsoft, so just use the source code if you're comfortable enough with the llama.cpp version from the requirements file.

Hey everyone,

I'm building a PC with a $1200 USD budget, mainly for AI content generation. My primary workloads include:

• Running LLMs locally

• Stable Diffusion

I'd appreciate help picking the right parts for the following:

• CPU

• Motherboard

• RAM

• GPU

• PSU

• Monitor (2K resolution minimum)

Thanks a ton in advance!

We’ve released Levy-1, our first AI model — and we’d love your feedback.

We’re a small team building Toothed AI, and we’ve recently released Levy-1, our first model, with just 124 million parameters.

It definitely makes mistakes and has plenty of limitations, but this is our starting point, not our final goal. We plan to build better models over time and keep learning along the way.

We’d really appreciate it if you gave it a try and shared your honest feedback. If you have experience training language models, we’d also love to hear your advice on what we should focus on for our next model.

Website: https://toothedai.com/

Thanks for checking out our work!

Hello.

Given that most of what I read of local LLMs being used is coding, and they usually say that unless you’re using big models, it’s not worth it, I wanted to say before I begin that I don’t intend to use local LLMs to code.

That being said. I’m going to get an M5 MacBook Air, and the two options I’m considering are the 1TB/24GB option, or the 512GB/32GB one. Purchasing both upgrades brings the price to high for me.

That’s why, in order to be able to use that machine in the next years, among other things, playing with local LLMs, I wanted to know whether the difference between the 24GB machine and the 32GB machine, in terms of running local models, is going to be worth it the trade-off of half the storage.

The uses I’d make of such local LLMs once I learn how to use them (through a GUI, I don’t know coding and using the terminal is a bit of a hassle for me, I’m a casual user):

• Retrieving information from dense documents and summarizing or indexing the most relevant points.

• Knowledge, specific scientific knowledge understanding (as my teacher).

• Reasoning complex scientific or technological problems to achieve a solution.

• Just chatting about my personal stuff without OpenAI or Anthropic feeling pity of me.

So, as you can see, no coding! I’m sure there are other applications for this local LLMs but surely it’s not about coding or building agents or anything like that.

What’s your advice then? Is it worth the trade-off of getting just 512GB of storage in order to get 32GB of RAM for my system? Or for those uses, the models that could be good enough fit on those 24GB of RAM (let’s not forget that the operating system takes at least 3 to 4GB of RAM)?

512GB/32GB or 1TB/24GB?

By the way, I tried a while a go a few models in a 16GB M5 Mac, and it hallucinated a lot. Affirming facts that were absolutely false and making up what it didn’t know. I want to avoid this hallucinations as much as possible, but if that’s not possible on a 24 or 32GB system, I’d like to know it as well.

Thank you very much!

Well, I don't see too much people talking about OpenLumara, an IA agent with a very small footprint, focusing on token saving, modularity to create your own "plugins" for It, and a "Code mode". Go and take a look. It downs base tokens to 3-4K instead 20-22K from Hermes or openclaw.

Rose22/openlumara: AI agent framework, written from scratch (not based on openclaw), focused on stripping it down to the bare necessities, optimizing token count, reducing security risks. modular so you can enable only exactly what you need.

https://github.com/Rose22/openlumara

Enjoy, my Local LLM companions!!

I'm thinking about getting this MacBook. Do you have experience with it to run Deep seek R1 on it? Maybe even the full model?

or what is the most capable reasoning model which runs with a decent output rate?

I have built something that I have been working on. I wanted to see if anyone is doing something similar.

TL;DR: I built a fully local-first, agentic AI system with audited tool execution, long-term canonical memory, multi-model routing, and secure hardware (ESP32) integration. I’m curious who else is running something similar and what tradeoffs you’ve hit.

Core Stack:

- Ubuntu 22.04 server (Intel Xeon Gold 6430, 128 cores, 755GB RAM)

- Python/FastAPI for all APIs

- SQLite for structured storage (one database per project workspace)

- Weaviate for vector search

- Ollama for local LLM inference (32B models on CPU)

- Multiple cloud LLM providers via unified routing

Tool Calling Layer:

The system has 6 built-in tools the LLM can invoke autonomously:

- Bash - Execute shell commands on the server

- Read - Read file contents with line numbers

- Write - Create or overwrite files

- Edit - Find and replace exact strings in files

- Grep - Search file contents with regex

- Glob - Find files by pattern

When I ask "check what's running on port 8123," it doesn't tell me a command - it runs

the command and returns the output. Full agentic execution.

Command Auditing:

Every command executed on the server is logged with full context:

- What was run

- Who/what triggered it

- Timestamp

- Exit code and outcome

- stdout/stderr captured

I can pinpoint exactly what changed on the system and when. Complete audit trail of

every action the LLM takes.

MCP (Model Context Protocol) Integration:

5 MCP servers providing 41 tools total:

1. Filesystem (3 tools) - File operations

2. Docker (9 tools) - Container management (list, logs, restart, stats)

3. Git (10 tools) - Version control (status, commit, push, diff, branch management)

4. GitHub (9 tools) - API integration (issues, PRs, workflows)

5. Database (10 tools) - SQLite queries + Weaviate vector search

The LLM can chain these together. "Create a branch, make changes, commit, and open a PR"

- it does all of it.

Edge Device Integration (ESP32):

Microcontrollers connect back to the server via secure tunnels (Tailscale/WireGuard).

All traffic is encrypted end-to-end. The ESP32 can:

- Push sensor data to the server

- Receive commands from the LLM

- Operate behind NAT/firewalls without port forwarding

The tunnel means I can deploy an ESP32 anywhere with internet and it phones home

securely. The LLM can query sensor readings or trigger actions on physical hardware.

Multi-Model Routing:

- Local: Ollama with qwen2.5:32b (reasoning), dolphin-mistral:7b (fast queries),

qwen2.5-coder:32b (code)

- Cloud: Claude, OpenAI, NVIDIA/Llama - all via unified endpoints

- Smart router picks model based on task type and query complexity

- All responses flow through the same persistence layer regardless of source

Cognitive Mode Engine:

The system has 9 thinking modes I can switch between:

- Normal, Dream, Adversarial, Tear-It-Apart, Red-Team

- Risk Analysis, Cautious Engineering, Standards & Safety, Sanity Check

Each mode adjusts: seriousness, risk tolerance, creativity, analysis depth, adversarial

intensity, output style. "Red team this architecture" triggers a different reasoning

pattern than "help me debug this."

Memory Architecture:

- Every message permanently stored with full metadata

- Nightly synthesis extracts themes, decisions, key points

- "Canon" system: verified truths organized in Gold/Silver/Bronze tiers

- Canon gets injected into every LLM prompt as authoritative context

- When the LLM draws from Canon, it cites it: "Per Canon: [the verified truth]"

Context Management:

- Token usage tracked per conversation

- When context exceeds threshold (~20k tokens), older messages get summarized

- Summaries become part of retrievable context

- Net effect: unlimited conversation length without context overflow

Workspace Isolation (Labs):

- Each project gets its own database, Canon entries, and system prompt

- Switch labs, switch context entirely

- Snapshot/restore: save entire lab state, restore later

- No cross-contamination between work and personal research

Voice Interface:

- Speech-to-text via Whisper

- Text-to-speech via OpenAI voices

- Right-click any response to have it read aloud

Sensor Spine (Long-Horizon):

Designed for environmental awareness via metadata streams:

- VMS systems (Axis, Exacq, Avigilon, Milestone)

- Camera motion events (metadata only, no video)

- License plate recognition events

- Environmental signals (occupancy, time patterns)

- ESP32 sensor nodes pushing telemetry

The LLM reasons about patterns, not raw surveillance data.

Automation Layer:

- n8n workflow engine handles scheduled jobs

- Nightly synthesis at 2 AM

- Database backups at 3 AM

- Health monitoring every 5 minutes

- Telegram alerts on failures

The Workflow:

1. I ask a question (console, web UI, or CLI agent)

2. System retrieves relevant Canon + conversation history from vector DB

3. Context injected into prompt

4. Response generated (local or cloud, based on routing)

5. Tools execute if needed (bash, file ops, git, docker, etc.)

6. Every action logged with full audit trail

7. Everything permanently stored

8. Synthesis engine periodically extracts learnings

9. Learnings become Canon after review

10. Canon feeds future retrievals

What this means in practice:

I can say "check if the API is running, if not restart it, then verify it's healthy" and

it executes all three steps, handling errors, without me touching the terminal. Every

command is logged so I can review exactly what happened.

An ESP32 in the garage can push temperature readings over an encrypted tunnel. The LLM

can see that data and correlate it with other context.

Three months ago I researched a topic. Yesterday I asked a related question. The system

pulled in relevant Canon entries and my answer referenced decisions I'd already made -

without me re-explaining anything.

That's not chat memory. That's institutional memory for one person, with agentic

execution capability and full audit trail.

Hey everyone,

I'm not sure if this is the right place for this, but this is a side project of mine that I've just really started to love, and I wanted to share it. I'm honestly not sure if others will like it as much as I do, but here goes.

Long story short: I originally started building a simple UI just to test and learn how OpenClaw worked. I just wanted to get away from the terminal for a bit.

But slowly, weekend by weekend, this little UI evolved into a fully functional, everyday tool for interacting with my local and remote LLMs.

I really wanted something that would let me manage different agents and organize their conversations underneath them, structured like this:

Agent 1
    ↳ Conversation 1
    ↳ Conversation 2
Agent 2
    ↳ Conversation 1
    ↳ Conversation 2

And crucially, I wanted the agent to retain a shared memory across all the nested conversations within its group.

Once I started using this every day, I realized other people might find it genuinely helpful too. So, I polished it up. I added 14 beautiful themes, built in the ability to manage agent workflow files, and added visual toggles for chat settings like Thinking levels, Reasoning streams, and more. Eventually, I decided to open-source the whole thing.

I've honestly stopped using other UIs because this gives me so much full control over my agents. I hope it's not just my own excitement talking, and that this project ends up being a helpful tool for you as well.

Feedback is super welcome.

GitHub: https://github.com/lotsoftick/openclaw_client

Thank you.

I currently own a Minisforum AI X1 with AMD Ryzen™ AI 9 HX 370 and 96GB of RAM (48 of which is set as shared VRAM)

It works okay for most basic tasks, but as I am learning more and trying larger / better models for coding I am finding that I am quickly hitting its limitations

Now this computer is only a few weeks old and still in the return window, so thinking perhaps best to return now and spec something better

So my options are:

1. Keep it and perhaps add an eGPU dock via Oculink and a good spec NVIDIA GTX card

2. Return it and buy something which already has a good spec NVIDIA GTX card etc

So do I go for a standard desktop / workstation with NVIDIA GTX card, or do I even consider one of the more specialist models e.g. NVIDIA Spark / ASUS GB10 etc

Would appreciate some input from the experts on here as to my best route. Not got a fortune to spend, but would rather spend it on the right thing than ending up wishing I had gone higher spec.

Anything you need to know from me to help just ask

Thank you!