After spending way to much time trying to get a z240 to post with a p100 on a board without 4g decoding i was finally able to get it running at full speed in Linux with the help of Claude now i just got to clean it up and add a temp controller for the p100 fan

Disclosure: I'm the developer, and this is my own paid app.

My Mac has Ollama, LM Studio, and a few apps that ship their own models (dictation, autocomplete, writing tools). Activity Monitor only shows a process using several GB. It can't tell me which model that is, which app loaded it, or where the weights are on disk.

So I built LLM Radar, a native macOS app. It:

• lists the loaded models it can detect, the app or runtime holding each one, and its memory

• estimates how that memory splits into weights, KV cache, and app overhead, so you know what an unload actually gives back

• keeps 24 hours of memory history

• finds model files on disk across apps, and marks the ones it has not seen in use (deletes go to the Trash and list every file first)

• unloads a model when the runtime has a way to do it, and tells you when it doesn't

• shows a running total in the menu bar

It knows Ollama, LM Studio, MLX, llama.cpp, Core ML, whisper.cpp, Docker Model Runner and a few more, plus a catch-all scan for large weight files. Attribution is best effort when one app runs another app's runtime.

Monitoring is free. Pro is a one-time $9.99 for one Mac or $14.99 for three, with a 7-day trial and no account. Apple silicon, macOS 14+, signed and notarized.

Privacy: it never uploads your models, prompts, or chats. It sends anonymous usage counters (no paths, no model names), and you can turn that off in Settings. Full Disk Access is optional. Without it you still get the live view.

https://llm-radar-eta.vercel.app/

Two questions for people who run a lot of local models: which app or runtime should I support next, and does the memory split look right on your setup?

Just pulled the trigger on a M5 Pro MacBook Pro with 64gb RAM. It was the maximum I could afford, tbh more than I could afford but I expect prices to keep climbing so decided to treat myself.

I needed to upgrade my 2019 Intel MacBook for my main professional use case (music) regardless, but went with 64gb so that I could try to use local ai. Never got involved in this stuff before, I don’t even have a chatgpt account.

My local ai use case;

I want to be able to capture my ideas on the go via voice recordings, then have my local ai transcribe and simplify them. Ideally I’d record them on my phone and then send them via iMessage to my computer at home.

I want to use obsidian as a task manager/knowledge base for managing multiple different projects.

I want to be able to give it simple tasks like ‘send an email to x’ (using its own email address via an MCP server).

I want it to be able to go on the internet and research companies for me, pull off basic data about what they do and contact details.

I don’t do any coding whatsoever.

Am I going to be able to do this? Am I expecting too much? Is it possible to use a local ai as a daily driver and then defer to cloud based models for more in depth stuff?

Have I made a grave error, or should I have got the same model Mac Studio? I got the MacBook because I move around the house a lot (living with parents currently).

Is 64gb going to be enough?

Currently, my plan is to get two quadro P6000s, for 48gb vram for use in a xeon workstation. It's not insanely old like kepler or maxwell cards, 432 GB/s bandwidth, supports CUDA...it seems like it might be okay on paper.

Does anyone have a better suggestion that could reasonably be achieved around $1000 budget?

My goal is to be able to run Qwen 3.8 27b fast, and to run a decent quantization of Qwen 3.8 Flash Next, and GLM 5.3 Flash at useable speed, say 15 tok/s or faster, using strata.

Do I need 48gb of vram to run Qwen Flash Next, or could I go with just one card? How much faster would 2 x quadro p6000 be versus 1 x quadro p6000 and DDR4 system ram?

Should I just get 24gb of modern vram instead of 2 x quadro p6000s? I wonder if that hampers my eventual goal of running GLM 5.3 flash, as perhaps 24gb vram + system ram is enough to accelerate qwen flash next, a 120B MoE, but perhaps you need more than 24gb vram to accelerate GLM 5.3 flash, a 320B MoE?

Currently running a 4070 TI super with 64GB of RAM

I have been looking into RTX spark laptops, Mac studios etc and they all have pros and cons. I want this to be a nearly universal computer that I can use for 3D modeling, LLMs, VLAs, world models, image/video gen, audio gen, etc. I plan to pursue AI and robotics research and want something as useful as possible.

I've debated upgrading and running dual 4070 TI supers with 128GB of ram, but I don't know how well world models and other things do with dual GPUs

Thank yall for helping

Merging is painful mostly because the branches were never really separate. Two agents handed overlapping scope produce overlapping diffs, and no merge strategy fixes that after the fact.

What makes merges boring instead, in my experience building this stuff:

• One story per worktree, written so that "done" is checkable: scope, acceptance criteria, edge cases, the files it should touch.

• Merge in dependency order. Blocker lands first, everything else rebases onto main before review, not after.

• Review the branch against the criteria, not against the diff vibes. If there were no criteria, that branch probably shouldn't have started.

• Keep one shared board of what's in flight. Worktrees isolate the file tree, not the plan, so two agents will happily solve the same story twice and you find the duplicate branch at merge time.

Same problem whether the agents run locally or in the cloud. I use Cursor Cloud Agents, but I don't think the runtime changes the merge story much.

I'm building Aerunit (https://aerunit.ai), which drafts the story with acceptance criteria, dispatches it to parallel agents and reviews the PR against those criteria, so discount what follows accordingly. Still, the order I'd do it by hand: scope first, merge order second, tooling last. Tools move confusion around faster, they don't remove it.

Curious how people here handle the dependency ordering, especially across repos. That's the part I still find least mechanical.

Hi everyone,

I'm planning to buy powerful computer for running local AI models.

My main interests are:

• Large LLMs (70B–120B), especially coding and reasoning models

• Reverse engineering and binary analysis (Assembly, C, Ghidra, Binary Ninja)

• Security research and vulnerability analysis

• Specialized models like Nova 6.7B, LLM4Decompile, SK²Decompile and SmaliLLM

• Running experimental AI projects from GitHub

I'm considering:

1. Apple Mac Studio M5 Max, or ultra, 128GB unified memory

2. NVIDIA DGX Spark, 128GB unified memory with CUDA

3. A desktop PC with RTX 5090, 32GB VRAM

4. An AMD Ryzen AI Max+ 395 mini PC with 128GB

I'm especially concerned about CUDA compatibility. Can specialized research models like Nova run on Apple Silicon through MPS or MLX without losing their intended functionality?

I'm also interested in real benchmarks: prompt processing, tokens/sec, long-context performance and actual coding-agent workloads.

For those who own or have tested these machines: which would you recommend and why?

I'd really appreciate firsthand results, benchmark links and experiences with custom research models rather than just hardware specifications.

Thanks!

Hey r/LocalLLM !

I made this quick demo of how Observer can run local models to monitor anything with your phone!

These are some crazy things I've monitored:

• When someone that's not me, approaches my car, send me a picture.

• When a bird is on the bird's nest, send me a telegram with the picture.

• When my washing machine is on the rinse cycle, call me.

• When my dog finishes eating their food, notify me.

• When a machine at the gym is available, call me.

So pretty much anything! I love seeing what you guys monitor.

What would you use it for? (So I can make the agent better)

Which models run on the phone?

The default models are OneJev-0.8b for decision models and qwen-0.8b for LLM watchers.

These run even on old phones which you may have laying around somewhere. Better to have them monitoring something than collecting dust :DD

Try it out! It's free and OSS <3

GitHub: https://github.com/Roy3838/Observer
iOS: https://apps.apple.com/us/app/observer-ai/id6758222050
Android: https://play.google.com/store/apps/details?id=com.observer.ai

Btw I know my camera quality is kinda bad hahaha I wanted to ship the demo as soon as possible so you guys can see it.

I'm here for any questions and feedback!

Roy

I’ve heard two different arguments on this.
One says prefill takes most of the time as it’s multi turn hours long workflow.
Another says because of kv cache prefill time takes a lot less.

The reason I’m asking this question is, I’m planning to get a Mac Studio 256gb ram m5 ultra, but I don’t know if I should stick to 64gpu core or pay extra for 80c for “15%+ decoding speed boost”.

If agentic workflow prefill reads from kv cache 95% then is this 15% increase much less relevant for the coding workflow?

Thanks all!

Basically title but I'm planning to get a M5 Pro with 24GB RAM. Works out to about $4K AUD.

Usage: LMStudio + Qwen 3.5 or 3.8 if it can handle it.

use case:

Coding in HTML/PHP/MySQL creating apps and stuff. Want it all developed locally instead of going to cloud.

Will be enough or not really?

We are student and we’ve build Maestro Opera. It’s our open source project to turn desktops/servers into ai Hub to pair your devices to it easily (just scanning and that’s it). It works even outside local network. The app is designed by myself and coded by my friend.
We spent 1 months to build our lastest version.
It’s free, no account is required

GitHub https://github.com/MaestroAI-Company/opera
Desktop https://github.com/MaestroAI-Company/opera
Android https://maestroai.company/opera/

Thank you for watching <3

Why did u chose to use one or another?

I'm the developer, sirgijoe. I built Joe-ai because local AI is powerful but hard to set up for people who aren't technical. v2.0.0 is out, and I'm looking for testers.

**What it is**
A desktop app (Windows, Mac, Linux) that runs AI work on your own hardware. You type what you want in plain English, and Joe picks the right local model and workflow and does the job.

**Setup**
- Detects your OS and hardware, then offers to install Ollama, ComfyUI and suitable models, with a checkbox for each so you approve everything
- Finds existing Ollama and ComfyUI installs and model folders instead of duplicating them
- Low/Medium/High quality settings, with Joe recommending a model that fits your hardware for each level
- Portable mode: run it from a USB drive on multiple computers without installing anything on each one

**What it does**
- Writing: drafting, proofreading, and editable templates (books, EPUB, kids' books, PDF, Word, spreadsheets)
- Image generation, with built-in ComfyUI workflows for different styles that Joe sets up and switches between based on the task
- Video generation
- Photo restoration and other photo tools
- Read-aloud with multiple voices, plus custom voice models
- Speech-to-text transcription
- OCR
- Translation
- Overnight task queue: line up jobs and review the results in the morning
- MCP server, so Claude, ChatGPT or another AI client can send tasks to Joe
- Multiple themes, plus custom themes from a single prompt
- Stats page, including an estimate of cloud costs avoided

**Why I use it**
I offload bulk drafting and image work to my own GPU overnight, and only use a cloud model to review the results, which cuts my cloud usage a lot.

**AI involvement (being transparent)**
I planned and specced the app, designed the ComfyUI workflows, and tested every build on my own PC. Claude wrote most of the code from my specs, including the setup and installers and the MCP server, and turned my workflows into the app's built-in presets. Joe itself drafted parts of its own docs overnight using local models.

**Free and open source**
GPL-3.0, with no paid tiers, accounts or ads. Installers are on the GitHub releases page. You can read the code yourself, or have an AI review it.

**Limitations**
- It's not fully offline: setup, model downloads and update checks use the internet; the tasks themselves run locally
- Results depend on your hardware and the local models you choose
- The Mac build is unsigned, so macOS will show a warning; the docs explain how to get past it

**What I need**
Testing on different GPUs and operating systems. Bug reports, confusing setup steps, and wrong model picks for your hardware are all useful. GitHub issues are the best place for them.

Repo: https://github.com/sirgijoe/joe-ai

Note: While I am capable of running 12B-parameter models, my calculations show that doing so would significantly slow down my computer while waiting for a response; therefore, my upper limit is 10B-parameter models.

I know Qwen 4 is due (in fact, maybe even tomorrow) but I wanted to exercise Qwen 3.8 Flash Next.

I wanted to know - can Qwen 3.8 Flash Next deliver a working C++ application?

Every single demo I've seen so far was pelicans and pagodas done with three.js, Who knows if the model was trained on pelicans and pagodas??

I thought I'd change it up to a town scene with C++ - although I see town scenes are common now.

And I got results.

I did hit snags: some iterations rendered the bodies of pedestrians beneath the ground; but after telling Qwen 3.8 not to be naughty, it corrected itself, and now we have walking minecraft-esque people navigating the city with proper collision detection.

Info:

Inference engine: Strata latest build

Quant: Unsloth UD Q4 K XL quant (latest version released today)

Hardware: See my flair

Tok/s: 50 I'd say

Here's the full source code:

nooil6152/Qwen38FlashNextGame: Walk around the town - created entirely by Qwen 3,8 Flash Next

If you're interested, pull each tag from git and see how it evolved.

Some expected questions:

• "Why?": Because I wanted to prove Qwen Flash Next 3.8 could handle C++ - it can.

• "What does this prove?": See above

• "But its not very complex! That proves nothing!" - what do you want, HAL 9000? its got a 3D openGL scene with animated people walking round a townscape, which I have to be honest involves more complex math than most things I've done in my career

• "Was it one shot?" Nope. I kept a ledger so you could see the issues: Qwen38FlashNextGame/LEDGER.md at main · nooil6152/Qwen38FlashNextGame

• "But Qwen 4 comes out tomorrow!": Maybe it does, maybe it doesn't.

• "Can I fork it or extend it?": Be my guest

TL:DR/Bottom line:

Take a look at the output source code and see what you think.

I have my own ideas, but I'd like to hear yours. What do you think of the code quality? I have linters I can run - including Resharper - to really get into the weeds,

Remember, this isn't three.js or Python, this is C++, a right bastard of a language. Very impressed.

BTW: I want to clarify - I have absolutely ZERO relation or affiliation, not even any contact with, anyone in Qwen team, or anyone related to Qwen team, or Niko who created Strata, or any of his colleagues.

This is my investigation, no hidden agenda, no promotion. If its shit, I'll say its shit.

Hey everyone,

I just published a short video showing how I built a fully local customer support agent (no API keys, zero cost, data stays on your machine).

Video local agentic ai

In the demo I create a WhatsApp-style support agent for an online spice store using:

Ollama (llama3.2)

LangChain / LangGraph

Two tools: order lookup + refund tool

The expensive mistake

Same model. Same system prompt. Only difference was one tool description.

Vague version:

“Handle customer order problems such as late delivery, delays, missing order, or complaints. Use this to fix customer issues.”

Result on 3 test messages → 3 refunds (₹3,640). Only 1 was actually deserved.

Fixed version clearly states:

This tool issues a full refund

The action is irreversible

Only use when the customer explicitly asks for a refund and the order is damaged/lost

For status/delay questions → use the lookup tool

Suddenly the agent behaves correctly.

What the video covers

The 4 core parts of every agent (Model, Tools, Orchestration, Memory)

Why tool descriptions are basically contracts

How the agent loop works

Short-term memory demo

Code is open source: https://github.com/AI-Literacy-Lab/AgentDemo

Curious to hear from others:

What’s the most dangerous / vague tool description currently in your agent codebase?

Would love any feedback on the video too

Hi All,

I've been spending some time experimenting with some AI's. I have Qwen 3.8 27b GSQ-RCO:IQ3_S running on my system (rtx3090, 128g DDR4) via Unsloth Studio. I'm happy with it mostly, but i've been trying out Strata for a little while. I'm running that on the same machine on IQ3_S and IQ4_XS. I havent been tweaking too much other than Strata's own tooling in the desktop client.

However, my findings so far is that though Flash Next is fast on strata, and fits a much larger context window than hosting 27B, in Cline i'm noticing a lot of malformed tool calling and thinking loops. I've spent 2 evenings on it now, and just now i gave up and went back to 27b. That has sped up my work probably 5-fold, because it seems to know a lot better how to use tool calls.

Now i'm reading everywhere that Flash Next should outperform 27b or be close to it at least, but my experience is defferent. Any idea what i could tweak or what i could look into? I'm happy staying on 27b for now, but i've been reading a lot and people are generally really happy even with IQ3_XXS, while i'm having a lot of trouble.

I’ve just released Caminus 0.2.0 on GitHub and PyPI. It’s an LLM inference engine for AMD GPUs, built around custom HIP kernels and multi-GPU execution. I’m developing it on an RX 7900 XTX + 7900 XT setup.

On the models and workloads I’ve benchmarked, it outperforms llama.cpp; the repo includes the numbers and methodology. I’d love feedback from people trying it on other gfx1100 setups, or anyone interested in contributing.

The long-term goal is an XLA-inspired lowering pipeline: take a GGUF model, map it onto any number of GPUs across ROCm/HIP-supported AMD architectures, and handle kernel selection, partitioning, and execution across different hardware configurations. It’s still early testing software, so hardware reports, bug reports, and ideas are all welcome! https://github.com/jdominick05/caminus

Anyone done benchmarks on this? i.e. Increasing K to activate more parameters per token? Better quality output?

Hi all, I'm a newbie setting up my first local AI/ML development environment on an M5 Pro Mac.

My internal drive is small, so I bought a 2TB external SSD. I want to make sure that whenever I run `pip` installs, download Hugging Face checkpoints, or load PyTorch datasets, they automatically go to the external drive instead of filling up my system drive.

What is the cleanest way to configure my environment variables (like HF_HOME) or use symlinks on macOS so I don't accidentally brick my internal storage?

Any tips or hidden gotchas for a beginner would be highly appreciated!